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Executive  Summary 

CAN  THE  AIR  FORCE  SOLVE  ITS  SPARES  FORECASTING  PROBLEM? 

The  credibility  of  Air  Force  estimates  of  future  spare  parts  requirements  has 
suffered  a  great  deal  in  recent  years.  Radically  changing  estimates  for  the  same  year 
are  one  of  the  main  reasons.  Since  1982,  for  example,  official  Air  Force  estimates  of 
the  FY87  requirement  have  changed,  either  up  or  down,  by  roughly  $1  billion  a  year. 
Changes  in  the  planned  operating  program  are  not  enough  to  explain  the  variations. 

The  Air  Force  Logistics  Command  (AFLC)  has  developed  a  regression  model, 
the  Air  Logistics  Early  Requirements  Technique  (ALERT),  to  improve  Program 
Objective  Memorandum  (POM)  forecasts  of  funding  requirements  for  reparable 
spares.  ALERT  attempts  to  predict  the  output  of  the  D041  Recoverable  Item/Central 
Secondary  Item  Stratification  (CSIS)  system,  the  Air  Force  procedure  for  computing 
and  executing  reparable  spares  budgets.  ALERT  uses  an  early,  "first  look"  D041/ 
CSIS  estimate  of  the  first  POM  year’s  requirement  as  a  key  input  for  projecting  whet 
the  D041/CSIS  "last  look"  will  be.  Other  inputs  include  age  and  value  of  the  aircraft 
fleet,  which  serve  as  additional  predictors,  and  D04TCSIS  budget  estimates,  which 
serve  as  "history"  for  the  2  years  preceding  the  first  POM  year, 

In  spite  of  its  conceptual  appeal,  we  do  not  think  ALERT  by  itself  can  solve  the 
credibility  problem  for  spares  requirements.  The  system  upon  which  ALERT  relies 
for  key  inputs,  the  D041/CSTS  budgeting  system,  is  too  volatile,  making  it  virtually 
impossible  for  ALERT  to  achieve  the  stability  and  accuracy  needed  for  credibility. 
From  FY81  to  FY85,  for  example,  D041/CSIS  budgets  differed  from  D041/CSIS  last 
looks  by  at  least  $800  million  for  each  of  those  5  fiscal  years.  Given  this  volatility  in 
the  underlying  D041/CSIS  system,  it  is  not  surprising  that  ALERT  outputs  are 
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unstable.  Successive  ALERT  estimates  of  the  FY88  requirement,  for  example,  have 
been:  $5  billion  (spring  1984),  $3.4  billion  (spring  1985),  and  $2.2  billion  (fall  1985). 

As  a  test  of  ALERT’s  accuracy,  we  used  it  to  make  a  "projection"  of  a  known 
historical  value,  the  last  look  for  FY85.  Using  inputs  that  would  have  been  available 
at  the  time,  the  ALERT  projection  is  $3.8  billion.  This  overstates  the  actual  last  look 
for  FY85  by  $1.5  billion  —  an  error  of  65  percent. 

Since  ALERT  represents  a  reasonable  attempt  to  link  POM  forecasts  with  the 
underlying  D041/CSIS  budgeting  and  execution  system,  improvements  in  ALERT 
results  are  unlikely  until  the  volatility  in  the  D041/CSIS  system  is  reduced. 
Improving  the  D041/CSIS  system  is  itself  a  formidable  task.  Key  to  its 
accomplishment  is  the  Requirements  Data  Bank  (RDB)  project  now  underway  at 
AFLC.  By  providing  improved  access  to  and  control  of  the  spares  data  base,  the  RDB 
is  aimed  at  improving  AFLC’s  ability  to  compute  spares  requirements  and  execute 
the  spares  program.  For  the  RDB  to  help  with  POM  and  budget  forecasting, 
however,  the  Air  Force  must  also  be  prepared  to  use  the  RDB  to  actively  manage  the 
requirement,  in  addition  to  computing  it.  Some  changes  in  requirements  are 
unavoidable,  but  many  reflect  judgments,  decisions,  and  schedules  that  could  be 
altered  or  adjusted  in  the  interest  of  requirement  stability.  A  key  role  for  the  RDB  is 
to  facilitate  this  improved  management. 

It  is  also  important  for  the  Air  Staff  and  AFLC  to  each  maintain  a  clear  view  of 
the  other’s  job.  Recognizing  that  requirements  for  recoverables  will  always  exhibit 
some  volatility  and  that  the  requirements  system  must  be  able  to  track  change  if 
AFLC  is  to  execute  properly,  the  Air  Staff  must  allow  AFLC  as  much  flexibility  as 
possible.  AFLC,  on  the  other  hand,  should  recognize  that  the  requirements 
estimates  it  submits  to  the  Air  Staff  need  to  be  relatively  stable  if  the  Air  Staff  is  tn 
successfully  defend  requirements  in  the  programming  and  budgeting  process. 


The  Air  Force  is  contemplating  stock  funding  as  an  alternative  way  of  financ¬ 
ing  reparable  inventories.  POMs  are  not  prepared  for  stock  fund  peacetime  replen¬ 
ishment  requirements  -  only  budgets.  If  reparables  are  stock-funded,  therefore,  the 
POM  forecasting  problem  ceases  to  exist  in  its  present  form.  Spares  funding  would 
still  have  to  be  programmed  into  the  accounts  that  buy  from  the  stock  funds.  For  the 
first  POM  year,  however,  such  requirements  could  be  based  on  previously  computed 
stock-fund  budgets.  For  the  POM  outyears,  requirements  could  be  based  on 
aggregate  forecasting  methods  simpler  than  ALERT. 

Whether  or  not  reparables  are  stock-funded,  successful  POM  forecasting  is 
more  likely  to  be  achieved  through  closer  cooperation  between  AFLC  and  the  Air 
Staff  in  developing,  prioritizing,  and  controlling  spares  requirements  than  through 
development  of  increasingly  more  complicated  forecasting  techniques. 
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1.  INTRODUCTION 


Each  May,  at  the  end  of  the  OoD  programming  cycle,  the  Air  Force  and  other 
Military  Departments  submit  Program  Objective  Memorandums  (POMs)  to  the 
Office  of  the  Secretary  of  Defense  (OSD).  The  purpose  of  POMs  is  to  formally  identify 
requirements  and  aid  in  resource  allocation  in  the  DoD.  POMs  contain  estimates  of 
the  funding  needed  over  a  future  5-year  period  that  begins  2  years  after  the  POMs 
are  submitted.  In  May  of  1986,  for  example,  the  Air  Force  submitted  POM  88,  which 
described  funding  needs  for  the  period  FY88  through  FY92. 

As  part  of  the  programming  process,  the  Air  Force  must  develop  estimates  of 
the  aggregate  funding  needed  in  each  of  the  5  POM  years  to  replenish  and  augment 
peacetime  operating  stocks  (POS)  of  depot-level-reparable  secondary  items,  i.e., 
spares.1 

After  POMs  are  submitted,  the  first  year  of  the  5-year  POM  period  becomes  the 
budget  year.  For  example,  after  submitting  POM  88  for  the  years  FY88  through 
FY92  in  May  1986,  the  Air  Force  finalizes  a  budget  for  FY88  and  submits  it  to  OSD 
in  October  1986.  Like  a  POM,  a  budget  must  include  a  statement  of  the  funding 
needed  for  replenishment  of  POS  spares. 

For  smooth  allocation  of  resources,  POM  and  budget  estimates  of  aggregate 
spares  requirements  should  be  about  the  same.  However,  there  is  an  important 
difference  in  the  Air  Force  between  programming  and  budgeting  for  spares.  For 


lIn  this  study,  we  discuss  only  POS  requirements,  which  constitute  the  bulk  of  recoverable 
secondary  item  requirements  [e  g  ,  $2  6  billion  out  of  a  total  spares  requirement  of  $3,8  billion  in 
FY87  —  for  both  POS  and  War  Reserve  Muteriel  (WRMil  The  Air  Force  projects  WRM  require¬ 
ments  by  means  other  than  those  used  for  determining  POS  requirements  WRM  requirements 
tend  to  be  for  "kits"  und  "sets"  bused  upon  specific  acquisition,  basing,  und  deployment  programs, 
POS  replenishment  requirements  are  driven  more  by  historical  demand,  future  operating  activity, 
changing  pipeline  factors,  additives,  and  modification  and  modernization  programs, 
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budgeting,  the  Air  Force  begins  with  "bottom-up,”  item-specific  calculations  of  gross 
item  requirements  and  projected  asset  positions.  The  difference  between  gross 
requirements  and  available  assets  is  priced,  accumulated  over  all  items,  and 
stratified  to  show  how  assets  have  been  applied  and  where  net  requirements  still 
exist.  (DoD  supply  policy  for  secondary  items  requires  bottom-up,  item-specific 
stratification.)  After  stratification,  the  Air  Force  makes  management 
adjustments  -  at  both  the  item  level  and  the  program  level  -  for  final  preparation 
of  a  budget.  FOMs  are  different,  For  POM  estimates,  the  Air  Force  does  not  begin 
with  item-level  calculations,  but,  instead,  uses  methods  that  are  more  top-down  and 
"macro”  in  concept.2 

This  report  addresses  spares  programming  in  the  Air  Force.  It  does  so  in  the 
context  of  an  evaluation  of  ALERT  (Air  Logistics  Early  Requirements  Technique),  a 
macro,  weapon-system-level  procedure  developed  by  the  Air  Force  Logistics 
Command  (AFLC)  for  forecasting  reparable  spares  requirements  for  POMs.  The 
credibility  of  Air  Force  estimates  of  future  spares  requirements  has  suffered  a  great 
deal  in  recent  years.  ALERT  represents  an  attempt  to  address  the  problem  by 
improving  POM  forecasts  for  spares. 

Our  first  main  conclusion  is  that  ALERT  will  not  solve  the  credibility  problem 
for  Air  Force  spares  requirements.  ALERT  will  not  be  able  to  deliver  the  stable  and 
accurate  forecasts  that  are  needed.  The  reasons  lie  not  so  much  with  ALERT  itself  as 
with  the  underlying,  item-specific  budgeting  and  execution  system,  the  D041/CSIS/ 
transition  system.  "D041”  is  shorthand  for  the  Recoverable  Consumption  Item 

2The  longer  length  of  POM  forecasting  horizons  compared  to  those  for  budgets,  particularly 
for  the  POM  outyears,  makes  item-specific  methods  for  POMs  impractical,  Item-level  data  are  not 
always  available.  Even  when  they  are,  the  logistics  factors  for  secondary  items  (e  g  ,  failure  and 
demand  rates,  repair  and  resupply  times,  condemnation  rates,  application  levels  in  the  fleet,  and 
obsolescence  rates)  change  so  much  over  time  thut  item-specific  projections  become  unreliable.  (In 
fact,  we  will  see  that  even  though  budgets  have  shorter  forecasting  horizons,  they  suffer  from  the 
same  problem,  further  complicating  the  forecasting  problem  for  spares.) 
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Requirements  System.  CSIS  stands  for  Central  Secondary  Item  Stratification. 
"Transition”  refers  to  the  management  review  and  adjustment  process  that  follows 
item-specific  computations.  The  D041/CSIS/transition  process  is  the  means  by 
which  AFLC  determines  and  establishes  what  the  spares  requirement  is  -  both  for 
budgets  and  for  execution.  POM  forecasts  are  forecasts  of  future  results  of  the 
D041/CSIS/transition  system. 

Our  second  main  conclusion  is  that  unless  D041/CSIS/transition-based  esti¬ 
mates  of  future  requirements  can  be  made  more  stable  and  accurate,  the  credibility 
problem  for  spare  requirements  will  persist  -  no  matter  what  POM  forecasting 
methods  the  Air  Force  may  develop.  This  is  because  the  credibility  problem  is 
caused  far  mote  by  persistently  large  differences  between  budgets  and  final  execu¬ 
tion  requirements  for  spares  -  a  problem  caused  by  defects  in  the  budgeting 
system  -  than  by  poor  POM  forecasts. 

A  final  chapter  points  out  various  management  actions  the  Air  Force  can  take 
to  deal  with  the  spares  programming  problem,  rather  than  continuing  to  seek 
technical  solutions  in  the  form  of  forecasting  models.  The  chapter  describes  how 
taking  these  management  actions  would  mitigate  the  credibility  problem  for  POMs 
and  allow  the  Air  Force  to  concentrate  on  the  more  fundamental  and  important 
problem  of  forecasting  budget  requirements  for  spares, 


2.  UNDERSTANDING  THE  PROBLEM 


I 


THE  CREDIBILITY  PROBLEM 


The  problem  facing  the  Air  Force  in  its  estimates  of  future  spares  requirements 
is  both  technical  and  perceptual.  The  technical  part  of  the  problem  has  to  do  with 
structural  and  technical  difficulties  involved  in  forecasting  future  funding  require¬ 
ments  for  spares.  The  perceptual  part  of  the  problem  has  to  do  with  the  fact  that 
many  people  simply  do  not  believe  Air  Force  forecasts  of  those  requirements  - 
regardless  of  how  they  are  constructed.  Ultimately,  it  is  the  latter  problem,  the 
credibility  and  defen3ibility  problem,  that  is  the  real  problem  to  be  solved.  To  he 
useful,  technical  forecasting  methods  and  models  should  help  make  projected 
requirements  more  credible  and  defensible  in  the  programming  and  budgeting 
process.  To  properly  evaluate  forecasting  methods  and  determine  their  chances  of 
success,  therefore,  we  must  first  understand  who  has  a  problem  with  spares  forecasts 
and  why.i 

Concern  about  estimates  of  future  Air  Force  spares  requirements  exists  both 
within  and  outside  the  Air  Force.  Externally,  in  the  course  of  annual  DoD  Planning, 
Programming,  and  Budgeting  System  (PPBS)  reviews,  considerable  skepticism 
about  spares  requirements  has  surfaced  within  OSD.  In  February  1986,  for  example, 
acting  on  concerns  raised  by  the  Program  Analysis  and  Evaluation  Directorate  and 


'Ongoing  controversies  ubout  secondary  Item  prices,  particularly  for  conn  it  mahle  repair 
purts,  have  exacerbated  the  forecasting  problem,  In  addition,  the  Air  Force  hus  numerous 
initiatives  underwuy  (such  as  competition  udvocucy  and  breakout  progrums,  changes  in  contracting 
and  ordering  procedures,  and  re-pricing  progrums)  that  will  affect  forecasts  of  future  funding 
requirements  for  spares,  The  pricing  problem  for  spares,  however,  is  not  the  central  topic  of  this 
report,  We  are  concerned  with  the  problem  o (requirements  forecasting  overall -of  which  pricing  Is 
only  a  part,  Also,  our  anulysis  is  only  for  nonconsumable,  depot-levelrepuruble  secondary  items 
1  items  with  Expendubillty,  Recoverubllity,  Repuirubllitv  Cutegory  (KKRC)  code:  XD|.  Luter,  we 
will  say  more  about  how  changing  prices  contribute  to  the  forecasting  problem, 
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the  Assistant  Secretary  of  Defense  (Acquisition  and  Logistics),  the  Deputy  Secretary 
of  Defense  notified  the  Secretary  of  the  Air  Force  that  "the  Air  Force  spares 
forecasting  methodology  will  be  looked  at  critically  during  the  OSD  review  of  POM 
88. ''2 

Concern  has  also  been  raised  on  Capitol  Hill.  In  the  fall  of  1985,  the  House 
Appropriations  Committee  proposed  removing  $1.3  billion  from  the  FY86  Air  Force 
POS  replenishment  spares  budget,  largely  as  a  result  of  committee  concern  about  the 
validity  of  the  Air  Force’s  require;  ients  forecasts. 

Internally,  the  Air  Force  has  been  struggling  with  the  problem  at  least  since 
the  early  1980’s.  Several  Air  Force  studies  critical  of  requirements  forecasting  for 
reparables  have  been  published, 3  and  replenishment  requirements  have  been 
subjected  to  increasingly  critical  scrutiny  by  Air  Staff  Panels  and  Boards.  Ironically, 
ALERT  seems  to  have  done  more  to  fan  the  fire  than  put  it  out.  Since  1984,  the  Air 
Force  Comptroller  has  repeatedly  questioned  ALERT  products  and  methods,  and 
flag-grade  officers  at  both  the  Air  Staff  and  AFLC  have  been  subjected  to  a  large 
number  of  ALERT  comparisons,  studies,  analyses,  and  briefings  as  the  controversy 
has  continued. 4 

Why  don’t  people  believe  Air  Force  estimates  of  future  spares  requirements? 
Because  estimates  of  the  same  year's  requirement  tend  to  change  radically  from  one 
year  to  the  next  -  giving  the  impression  that  the  Air  Force  does  not  know  what 
future  spare  requirements  will  be.  Since  1982,  for  example,  official  Air  Force  POM 

'■^Memorandum  for  the  Secretary  of  the  Air  Force,  Subject:  "Air  Force  Replenishment  Spures 
Requirements,"  from  tho  Deputy  Secretary  of  Defense,  February  1986, 

3Seo;  SARER  PROVIDER  -ALPHA:  Programming  for  Aircraft  Replenishment  Spams 
(Headquarters,  L’.S,  Air  Force  L'.S.  Air  Force  Studies  and  Analysis),  December  1981;  and 
CORONA  REQUIRE:  An  Analysis  of  the  Aircraft  Replenishment  Spares  Acquisition  Process, 
Chairman,  General  Alton  D,  Slay,  L'.S  Air  F  orce  (Retired),  sponsored  by  Chief  of  Stuff  of  the  Air 
Force,  March  1983. 

■♦Indeed,  the  fact  that  the  controversy  over  spares  requirements  has  continued  is  prima  facie 
evidence  that  ALERT  cannot  and  will  not  solve  the  credibility  problem. 
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estimates  of  the  FY87  POS  requirement  have  changed,  either  up  or  down,  by  roughly 
$1  billion  each  year  (see  Figure  2-1  >5. 


The  problem  caused  by  changing  estimates  is  made  clear  in  the  February  1986 
memorandum  from  the  Deputy  Secretary  of  Defense  to  the  Secretary  of  the  Air  Force 
(cited  earlier).  The  memorandum  states  that  "large  fluctuations  in  spares  require¬ 
ments  ...  are  clear  indications  that  a  credibility  problem  exists." 


5The  Air  Force  has  changed  its  POM  forecasting  methods  for  POS  requirements  over  the  time 
period  addressed  in  Figure  2-1  The  estimates  in  POM  83  and  POM  84  were  based  on  u  coslper- 
flying-hour  method.  The  estimates  in  POM  85  und  POM  86  were  derived  from  the  POS  Spares 
Estimating  Model  (POSSEM),  developed  by  the  Air  Force  Comptroller  The  estimate  in  POM  87 
was  derived  from  ALERT,  Although  the  use  of  different  models  explains  some  of  the  fluctuation  in 
the  estimates,  this  does  not  mitigute  the  credibility  damage  that  has  been  done  More  important, 
there  are  structural  reusons  underlying  the  volatility  shown  in  Figure  2-1,  which  we  will  see  ure 
likely  to  cause  any  POM  forecasting  model  that  relies  on  D04l/CSIS/trunsition  to  exhibit  high 
levels  of  instability  -  even  if  the  same  model  is  used  from  one  year  to  the  next. 


How  is  it  that  POM  estimates  of  a  given  year’s  requirement  can  change  from 
year  to  year?  There  are  many  possible  reasons. 

Every  year  the  Air  Staff,  AFLC,  and  other  Air  Force  Commands  make  changes 
in  the  planned  flying-hour  program,  modify  force  structure  and  sizing  plans,  and 
adjust  schedules  for  modernization  and  modification  programs.  Because  these 
changes  affect  future  spares  requirements,  one  cannot  expect  POM  estimates  of  a 
given  year’s  requirement  to  be  identical  from  one  year  to  the  next.  The  problem  is 
that  these  changes  art  not  enough  to  explain  variations  as  large  as  those  reflected  in 
Figure  2-1. 

For  example  -  as  Figure  2-1  shows  -  the  POM  87  estimate  of  the  FY87  POS 
requirement  was  $2.6  billion,  down  more  than  $1  billion  from  the  POM  86  estimate 
for  FY87  of  $3.63  billion  -  a  reduction  of  more  than  27  percent.  Bu  tTable  2-1  shows 
that  neither  the  projected  fleet  size  nor  the  planned  flying-hour  program  changed  by 
enough  to  explain  such  a  large  reduction  in  the  requirement.  (Table  2-1  shows  air¬ 
craft  inventory  and  total  flying  hours  in  FY89.  FY89  is  an  average  procurement 
leadtime  beyond  FY87,  so  aircraft  inventory  and  flying  hours  in  FY89  reflect  the 
operating  program  that  is  supported  by  FY87  funding  for  spares.)  This  is  not  to  say 
that  either  the  POM  86  or  POM  87  estimate  for  FY87  was  "right.”  Rather,  the  two 
estimates  are  so  different,  without  understandable  reasons  why,  that  the  credibility 
of  both  estimates  suffers. 

It  is  also  true  that  funding  requirements  for  reparables  are  net  requirements, 
reflecting  the  difference  between  gross  requirements  and  available  assets.  As  a 
result,  a  given  percentage  change  in  either  direction  in  the  planned  operating 
program  will  often  produce  a  greater  percentage  change,  in  the  same  direction,  in 
the  requirement.  For  example,  a  sensitivity  analysis  done  for  the  Air  Staff  with  the 
LMI  Aircraft  Availability  Model  shows  that  a  10-percent  reduction  in  the  Air  Force 
flying-hour  program  planned  for  the  period  FY88  through  FY94  would  enable  the 
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TABLE  2-1.  OPERATING  PROGRAM  CHANGES  UNDERLYING  POM  86-TO-POM  87 
CHANGES  IN  THE  FY87  POS  SPARES  REQUIREMENT 


USAF/AFLC  AEROSPACE 
PROGRAM  DOCUMENTS 

USAF  AIRCRAFT 
INVENTORY 

IN  FY89 

USAF  FLYING-HOUR 
PROGRAM  IN  FY89 
(Hundreds  of  hours) 

PA86-1/K004  (Fall  198S) 

9,158 

38,685 

PA87-1/KQ04  (Fall  1986) 

9,062 

37,749 

Percent  change 

-  IVo 

-  2.4% 

Air  Force  to  reduce  FY87  POS  funding  of  $2,078  million  by  $305  million  (a 
14.7-percent  reduction),  without  degrading  aircraft  support.  This  degree  of 
sensitivity  is  not  enough,  however,  to  explain  how  program  changes  as  small  as 
those  listed  in  Table  2-1  could  produce  a  reduction  in  the  FY87  requirement  as  large 
as  the  one  that  took  place  between  POM  86  and  POM  87. 

Changes  in  estimated  acquisition  savings  (e.g.,  due  to  increased  competition), 
reliability  and  maintainability  improvements,  and  changes  in  plans  for  modification 
programs  will  also  affect  year-to-year  estimates  of  a  given  year's  funding  require¬ 
ment  for  spares.  Again,  these  changes  are  not  enough  to  explain  the  large  variations 
shown  in  Figure  2-1.  For  example,  POM  86  specified  a  total  of  $1.19  billion  in  class 
IV  (reliability  and  maintainability  improvement)  modification  programs  for  FY89. 
Such  modification  programs  generate  spares  requirements  in  the  form  of  nonrecur¬ 
ring  demand  for  replacement  parts.  In  POM  87,  the  FY89  class  IV  modification 
program  was  set  at  $1.43  billion,  an  increase  of  $240  million  in  the  planned  program. 
In  general,  one  would  expect  sucn  a  change  to  have  made  the  FY8?  spares 
requirement  go  up  from  POM  86  to  POM  87,  not  down  as  it  did. 

It  is  tempting  to  think  that  prior-year  funding  is  another  important  f  for 
influencing  estimates  of  a  given  year’s  requirement.  If  year  X  is  not  "fully  funded” 


(i.e.,  the  Air  Force  does  not  get  what  it  asked  for  from  OSD  and  Congress),  this  can 
have  a  significant  effect  on  the  next  estimate  of  year  (X  4-  l)'s  requirement,  due  to 
carry-over  of  unfunded  requirements  from  year  X  into  year  (X  +  l)’s  budget  request. 
This  would  mean  that  funding  decisions  external  to  the  Air  Force  -  which  take 
place  after  forecasts  have  been  made  -  may  be  another  reason  why  requirements 
estimates  vary  from  year  to  year.  This  is  an  important  point  and  one  that  must  be 
considered  when  examining  stability  problems  in  the  budgeting  system  for  spares.  It 
cannot  be  used  to  explain  fluctuations  in  POM  estimates,  however.  Outyear  POM 
estimates  are  "stand-alone”  estimates  that  assume  full  funding  of  prior-year 
requirements,  and  the  first  POM  year  can  include  carryover  only  if  the  Air  Force 
itself  elects  not  to  request  full  funding  for  the  (prior)  budget  year. 

Finally,  suppose  that,  because  of  the  combined  effect  of  the  many  changes 
between  one  POM  cycle  and  the  next,  fluctuations  as  large  as  those  in  Figure  2-1 
were  the  norm  and  had  to  be  accepted  as  a  fact  of  life.  This  raises  another  problem. 
The  purpose  of  programming  and  POMs  is  to  provide  for  consistent  and  coherent 
allocation  of  resources  within  the  DoD  planning  process.  To  do  that,  POM  forecasts 
must  provide  some  reasonable  level  of  accuracy.  If  "forecasts”  of  a  given  year’s 
requirement  can  normally  be  expected  to  change  by  more  than  $1  billion  from  one 
year  to  the  next,  allocating  resources  on  the  basis  of  those  forecasts  is  a  futile 
exercise. 

The  moral  is  this;  POM  forecasting  models  for  spares  are  useful  in  program¬ 
ming  to  the  degree  that  they  can  yield  reasonably  stable  estimates  of  future  require¬ 
ments  -  estimates  that  recognize  there  will  always  be  some  change  from  one  year  to 
the  next,  but  also  recognize  that  too  much  change  (independent  of  major  program 
changes)  is  self-defeating. 
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TECHNICAL  ASPECTS  OF  THE  PROBLEM 

As  noted  earlier,  there  are  technical  and  structural  difficulties  involved  in  fore¬ 
casting  spares  requirements  for  POMs.  These  difficulties  are  separate  from,  but 
contribute  to,  the  credibility  problem.  This  section  describes  the  technical  and  struc¬ 
tural  characteristics  of  the  forecasting  problem  for  spares. 

Figure  2-2  illustrates  the  forecasting  problem  for  POM  88.  POM  88  serves  as 
an  example  throughout  the  remainder  of  the  report. 


The  POM  forecast  for  FY88  illustrated  in  Figure  2-2  is  actually  the  last  in  a 
sequence  of  five  POM  forecasts  in  which  FY88  requirements  were  included.  The  first 
was  POM  84,  which  was  submitted  to  OSD  in  May  1982.  In  that  POM,  FY88  was  the 
fifth  year  of  the  5-year  POM  period,  FY84  through  FY88.  By  POM  88,  FY88  had 
become  the  first  year  of  the  5-year  POM  period.  The  instability  of  POM  estimates, 
noted  earlier  (Figure  2-1),  refers  to  instability  in  these  successive  POM  estimates. 
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The  next  thing  to  notice  in  Figure  2-2  is  that  for  the  Air  Staff  to  meet  a  May 
1986  deadline  for  submitting  POM  88  to  OSD,  AFLC  must  have  its  POM  88  esti¬ 
mates  to  the  Air  Staff  by  the  previous  fall.  Given  the  quarterly  schedule  for 
D041/CSIS  processing  (discussed  further  below),  this  means  that,  for  input  to 
ALERT  for  POM  88  estimates,  AFLC  must  rely  on  D041/CSIS  data  that  were  pro¬ 
duced  in  June  of  1985.  POM  forecasting  horizons  for  AFLC,  therefore,  are  more  than 
2  years  for  the  first  POM  year,  out  to  more  than  6  years  for  the  fifth  POM  year. 

The  dollar  symbol  in  Figure  2-2  is  a  reminder  that  POM  forecasts  are  projec¬ 
tions  of  funding  requirements.  Funding  enables  the  Air  Force  to  obligate  money  to 
contractors  and  suppliers  for  spares;  the  spares  themselves  do  not  arrive  until  a 
procurement  leadtime  (PLT)  later.  Thus,  in  terms  of  material  requirements,  AFLC’s 
POM  forecasting  horizons  are  from  4  to  8  years  into  the  future,  based  on  an  average 
PLT  for  recoverable  components  that  is  now  more  than  2  years  long. 

We  have  just  said  that  a  POM  forecasting  model  attempts  to  project  the  funding 
requirement  for  spares.  What  exactly  do  we  mean  by  the  "funding  requirement"? 
Put  another  way,  how  exactly  does  the  Air  Force  go  about  measuring  the  funding 
requirement?  The  answer  is  that  the  funding  requirement  for  a  given  year  is 
whatever  the  D041/CSIS/transition  system  says  it  is  in  the  "last  look”  it  takes  at  that 
year.  To  understand  what  this  means,  we  must  examine  further  the  way  the  D041/ 
CSlS/transition  system  operates. 

The  D041/CSIS  system  is  run  quarterly  in  June,  September,  December,  and 
March.  Full  transition  adjustments  to  the  results  are  normally  made  after  the 
March  and  September  computations  only.  Transition  adjustments  involve  a  con¬ 
siderable  amount  of  time-consuming,  nonautomated  interaction  (e.g.,  meetings  and 
management  reviews)  between  the  Air  Logistics  Centers  and  Headquarters  AFLC. 
They  are,  therefore,  normally  done  only  for  the  quarterly  computations  used  to 
c  ort  budgets  and  POMs,  that  is,  the  March  and  September  computations. 


The  June  computation  is  the  first  in  the  four  D041/CSIS  computations  each 
year.  It  projects  item  material  requirements  and  asset  positions  for  each  quarter,  out 
through  a  total  of  25  quarters  (6±  years).  The  first  quarter  is  the  "current  year” 
(CY).  The  next  four  quarters  are  the  "apportionment  year"  (AY).  The  following  four 
are  the  "budget  year"  (BY),  and  the  four  after  that  are  the  "extended  year"  (EY).  The 
September  computation  drops  the  first  quarter  (the  "current  year"  quarter  in  the 
June  computation)  and  looks  forward  24  quarters.  The  December  and  March  compu- 
tations  do  the  same,  looking  forward  23  and  22  quarters,  respectively. 

To  compute  item  funding  requirements,  the  D041/CSIS  system  must  back  off  a 
procurement  leadtime  from  the  time  when  item  material  requirements  exist. 
Because  there  are  items  in  the  data  base  with  a  3-year  (12  quarter)  procurement 
leadtime,1 8  aggregate  funding  requirements  for  the  whole  data  base  can  be  seen  only 
13  quarters  ahead  with  a  June  data  base  (25  quarters  minus  12  quarters).  Thus,  a 
June  computation  sees  funding  requirements  in  the  CY  (only  one  quarter  long),  the 
AY,  the  BY,  and  the  EY.  Similarly,  a  September  computation  sees  funding 
requirements  through  12  quarters  -  the  AY,  BY,  and  EY.  (The  CY  exists  for  the 
June  computation  only.)  December  projects  funding  reqi  rements  for  the  last  three 
quarters  of  the  AY,  and  all  of  the  BY  and  EY.  March  projects  for  the  last  half  of  the 
AY,  and  all  of  the  BY  and  EY. 

Because  it  is  the  first  scrubbed  D041/CSIS  computation  to  see  FY88,  the 
September  1985  computation  is  called  the  "first  look”  at  FY88.  The  "second  look”  at 
FY88  is  the  March  1986  computation  done  6  months  later.  The  last  look  at  the  FY88 
requirement  is  the  scrubbed  computation  in  March  1988,  6  months  into  actual 
execution  of  FY88.  It  represents  the  sixth  scrubbed  D041/CSIS  estimate  of  FY88 

•Mu  fact,  there  are  items  in  the  D041  data  base  with  procurement  leadtimes  (PLT)  greater 
than  3  years.  For  convenience  in  processing,  however,  D041  code  imposes  a  cutoff  of  3  years  as  a 
maximum  allowable  PLT. 


requirements.  It  is  based  on  the  results  of  the  first  6  months  of  actual  execution  in 
FY88,  coupled  with  a  requirements  projection  for  the  last  6  months. 

To  review:  AFLC’s  spares  forecasting  problem  for  POMs  is  to  provide  the  Air 
Staff  each  fall  with  projections  of  what  the  D041/CSIS/transition  last-look  estimates 
will  be  of  funding  requirements  in  each  of  the  5  POM  years.  For  POM  88,  therefore, 
ALERT  must  forecast  what  the  last  look  will  be  for  each  of  the  years  FY88,  FY89, 
FY90,  FY91,  and  FY92.  To  do  this,  AFLC  must  rely  on  an  unscrubbed  (i.e.,  non- 
transitioned)  D041/CSIS  computation  from  the  prior  June  (e.g.,  June  1985  for 
POM  88).  The  June  computation  is  the  first  D041/CSIS  run  to  see  the  first  POM  year 
(which  it  sees  as  the  extended  year),  and  it  is  the  only  D041/CSIS  run  that  produces 
results  in  time  for  AFLC  and  ALERT  to  meet  Air  Staff  deadlines  for  initial  POM 
estimates. 


3.  BUDGET  AND  EXECUTION:  THE  BEHAVIOR 
OF  THE  D04 1/CSIS/TRANSITION  SYSTEM 


INTRODUCTION 

The  previous  chapter  shows  that  POM  forecasts  for  spares  are  attempts  to 
project  what  another  system,  the  D041/CSIS/transition  system,  will  say  the  spares 
requirement  is.  The  next  chapter  describes  how  ALERT  attempts  to  do  this  -  using 
data  from  various  sources,  including  the  D041/CSIS/transition  system.  Clearly,  this 
system  -  the  budgeting  and  execution  system  -  is  central  to  the  requirements  fore¬ 
casting  problem  for  spares  in  the  Air  Force.  This  chapter  describes  how  the  D041/ 
CSIS/transition  system  behaves. 

We  have  two  main  points  to  make.  First  we  show  that,  from  the  time  a  budget 
estimate  is  made  to  the  time  a  "last  look”  is  taken  at  a  given  year's  requirements,  the 
D041/CSIS/transition  system  exhibits  as  much  instability  in  its  estimates  of  the 
requirement  as  POMs  have  been  accused  of  doing.  In  the  next  chapter  we  will  see 
how  this  volatility  in  the  D041/CSIS/transition  system  taints  ALERT  projections, 
undermining  both  their  stability  and  their  potential  for  accuracy. 

Second,  we  will  show  that  changing  D041/CSIS/transition  estimates  -  more 
than  changing  POM  estimates  -  are  the  prime  cause  of  the  credibility  problem  for 
spares  in  the  Air  Force.  Thus,  attempts  to  improve  or  develop  new  POM  forecasting 
models,  even  if  judged  successful  by  some  other  criteria,  still  cannot  and  will  not 
solve  the  fundamental  problem.  Even  with  a  crystal  ball  for  POMs,  the  Air  Force 
will  (and  must,  under  established  DoD  supply  policy)  continue  to  make  and  execute 
Bpares  budgets  with  a  bottom-up,  item-specific  system,  like  the  D041/CSIS/tran3i- 
tion  system.  As  long  as  that  system  continues  to  be  volatile,  the  credibility  problem 
will  persist. 

i 

i 
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THE  ROLLER  COASTER  FROM  BUDGET  TO  LAST  LOOK 

Figure  3-1  compares  Budget  Estimate  Submissions  (BESs)  for  each  of  the  years 
FY81  through  FY85  with  last  looks  for  the  same  years.  The  Figure  shows  that  since 
the  early  1980's,  budgets  for  spares  have  been  extremely  poor  indicators  of  what  the 
Air  Force  eventually  decides  it  wants  to  spend  in  actual  execution.  Figure  3-1  is  the 
most  important  figure  in  this  report.  The  volatility  it  depicts  is  fundamental  to  why 
POM  forecasting  is  never  likely  to  be  successful.  Even  more  important  than  the 
implication  for  POM  forecasting,  however,  is  the  degree  to  which  the  budgeting 
system  -  the  D041/CSIS/transition  system  -  is  not  forecasting  correctly.  This 
failure  of  the  budgeting  system  is  the  prime  cause  of  the  credibility  problem  for 
spares  -  more  important  than  the  failure  of  the  POM  forecasting  system. 


All  the  data  in  Figure  3-1  were  obtained  from  official  Air  Force  Budget  Esti¬ 
mate  Submissions  to  OSD.  Because  the  message  of  the  figure  is  central  to  the  report 
and  key  to  the  arguments  that  follow,  the  source  and  meaning  of  the  data  in  the 
figure  should  be  well  understood. 

To  explain  the  data  in  Figure  3-1,  we  focus  on  one  year,  FY85.  Two  data  points 
appear  over  FY85.  The  point  on  the  solid  line,  $4,1  billion,  represents  the  total  POS 
budget  request  (POS  total  requirement)  for  FY85,  as  submitted  by  the  Air  Force  to 
OSD  in  October  1983  in  the  FY85  BES.t  That  amount  is  based  on  D041/CSIS/ 
transition  processing  of  a  March  1983  D041  data  base  by  AFLC,  followed  by 
interaction  and  negotiations  between  AFLC  and  the  Air  Staff  before  final 
submission  of  the  BES.  In  the  language  of  the  D041  calendar  described  in  the 
previous  chapter,  D04i/CSIS/transition  processing  of  a  March  1983  data  base  sees 
FY85  funding  requirements  as  extended  year  (EY)  requirements. 

The  point  on  the  dotted  line  above  FY85,  $2.6  billion,  is  the  total  POS  require¬ 
ment  for  FY85,  as  submitted  by  the  Air  Force  to  OSD  in  October  1985  in  the  FY87 
BES2  -  a  budget  submission  appearing  2  years  after  the  FY85  BES.  The  FY87  BES 
still  includes  an  estimate  for  FY85  because  every  BES  includes  a  statement  of  the 
requirement,  not  only  for  the  budget  year,  but  also  for  the  2  years  preceding  the 
budget  year.  The  difference  for  the  preceding  years  is  that  their  estimates  are 
derived  from  the  more  recent  D041  data  base  and  D041/CSIS/transition-system 
processing  underlying  the  new  budget.  The  $2.6  billion  amount  thus  reflects  the 
results  of  D041/CSIS/transition  processing  of  a  March  1985  data  base  by 
AFLC  —  the  last  look  at  FY85.  In  the  language  of  the  D041  calendar,  D041/CSIS/ 

United  States  Air  Force  Budget  for  FY85  Budget  Progrum  1500,  Aircraft  Replenishment 
Spares,  3010  Aircraft  Appropriation  (Summary  of  FY85  Budget  Requirements),  1  October  1983, 

p.  3. 

2United  States  Air  Force,  FY8T  Budget  Estimate  Submission,  Aircruft  Replenishment 
Spares- BP-15, 3010  Aircraft  Appropriation  (Requirements  Summary),  t  October  1985,  pp.  1-4. 


3-3 


transition  processing  of  a  March  1985  data  base  sees  FY85  funding  requirements  as 
apportionment  year  (AY)  requirements. 

In  similar  fashion,  the  two  data  points  over  each  of  the  other  Fiscal  years  along 
the  axis  in  Figure  3-1  were  obtained  from  BESs  that  were  separated  in  time  by 
2  years. 

Figure  3-1  shows  that  for  every  fiscal  year  from  FY81  to  FY85  (the  last  fiscal 
year  completed  at  the  time  of  this  report),  there  has  been  at  least  an  $800  million 
difference  (in  one  direction  or  the  other)  between  the  budget  estimate  for  the  year 
and  the  last-look  estimate  for  the  same  year. 

Just  as  for  POM  estimates  separated  in  time,  we  cannot  expect  budget  esti¬ 
mates  and  last  looks  to  be  identical.  Many  things  can  and  do  change  in  the  2  years 
between  the  assembly  of  a  budget  and  the  taking  of  a  last  look.  Further,  if  we  take 
the  traditional  view  that  the  D041/CSIS/transition  process  defines  the  requirement, 
we  cannot  really  ask  (as  we  did  for  POMs)  whether  the  changes  are  large  enough  to 
explain  such  large  fluctuations  in  the  requirement.  If  the  requirement  is  whatever 
the  D041/CSIS/transition  system  says  it  is,  the  changes  were,  by  definition,  large 
enough. 

The  problem  is  that  if  a  given  year’s  spares  requirement  estimate  really  is 
subject  to  change  by  $800  million  or  more  between  budget  and  execution,  it  becomes 
difficult  to  put  much  faith  in  annual  budget  submissions.  So,  the  questions  we  have 
to  ask  are:  What  causes  estimates  of  a  given  year’s  requirement  to  change,  and  is 
there  anything  the  Air  Force  can  do  to  monitor  and  (where  possible  and  appropriate) 
control  the  changes  to  bring  budgets  and  last  looks  closer  together? 

WHY  DO  D04 1/CSIS/TRANSITION  ESTIMATES  CHANGE  ? 

Changes  in  the  planned  force  size,  in  schedules  for  modernization  and  modifica¬ 
tion  programs,  and  in  the  planned  flying-hour  program  explain  in  part  why  D041/ 
CSIS/transition  estimates  of  a  given  year’s  requirement  will  change  from  year  to 


year.  As  we  have  seen,  however,  these  changes  are  generally  not  enough  to  explain 
fluctuations  as  large  as  those  recorded  in  Figure  3-1.  If  these  were  the  only  factors 
causing  estimates  to  change,  stabilizing  the  spares  requirement  would  mean  basing 
some  program  decisions  on  spares  effects.  Although  this  would  be  putting  the  cart 
before  the  horse  for  many  programs,  it  might  make  sense  in  some  cases.  Certainly, 
taking  spares  effects  into  account  when  making  program  decisions  (e.g.,  flying-hour 
program  decisions)  is  a  good  idea.  Air  Force  planning  should  be  flexible  —  but 
mechanisms  for  senior  management  review  must  exist  to  ensure  that  changes  that 
will  significantly  modify  the  budget  are  necessary  and  important  enough  to  justify 
those  modifications. 

What  about  funding  decisions  for  prior  years?  What  effect  do  they  have  on 
successive  estimates  of  a  year’s  requirements?  Unlike  POMs,  budget  requests  may 
include  unfunded  carry-over  requirements  arising  us  a  result  of  prior-year  funding 
decisions  by  OSD  and  Congress.  Unfunded  carry-over  requirements  are  in  addition 
to  ’’stand-alone"  requirements  computed  for  a  given  year.  Unfunded  carry-over 
supposedly  represents  real  requirements  that  continue  to  exist  and  continue  to 
require  funding.  This  means  that  funding  decisions  for  prior  years,  made  by  organi¬ 
zations  outside  the  Air  Force,  can  influence  the  size  of  the  total  budget  request  for  a 
given  year.  Even  within  the  Air  Force,  if  the  Air  Force  Comptroller  or  other 
elements  of  the  Air  Staff  elect  not  to  go  forward  with  what  AFLC  has  determined 
(using  the  D041/CSIS/transition  system)  to  be  the  budget  requirement,  next  year’s 
computed  budget  from  AFLC,  which  includes  transition  adjustments  made  by  AFLC 
management,  may  include  carry-over, 

The  budget  requests  in  Figure  3-1  include  both  kinds  of  requirements  — 
computed,  stand-alone  requirements  and  unfunded  carry-over  requirements  -  for 
each  year.  The  budget  figure  of  $4.1  billion  from  the  FY85  BES,  for  example,  is  the 
sum  of  a  computed  requirement  of  $3,3  billion  for  FY85  and  a  prior-year  unfunded 


carry-over  requirement  of  $0.8  billion  from  FY84  into  FY85.  We  have  compared  the 
$4.1  billion  figure,  rather  than  $3.3  billion,  with  the  last  look  of  $2.6  billion  for 
FY85. 

There  are  two  reasons  why  it  is  correct  to  compare  the  $4.1  billion  with  the 
$2.6  billion,  as  we  have  done.  First,  the  last  look  of  $2.6  billion  includes  any 
requirements  that  really  did  carry  over  from  FY84  into  FY85.  Thus,  since  the  last 
look  includes  both  computed  and  carry-over  requirements,  it  is  appropriate  to 
compare  it  with  the  total  budget  request  -  not  just  the  stand-alone  portion.  Second, 
the  $4.1  billion  request  in  the  FY85  BES  already  includes  and  takes  into  account 
funding  decisions  for  FY84  and  FY83.  In  October  1983,  at  the  time  of  the  FY85  BES, 
FY83  is  ending  and  FY84  is  starting,  and  funding  decisions  for  both  years  have 
already  been  made.  These  funding  decisions  have  been  taken  into  account  in  the 
BES’s  statement  of  the  FY85  requirement. 

Funding  decisions  that  can  help  explain  a  difference  between  a  BES  and  a  last 
look  are  funding  decisions  made  after  the  BES  is  submitted.  For  the  FY85  BES,  for 
example,  a  decision  after  October  1983  to  change  the  funding  in  FY84  would  help 
explain  why  the  last  look  for  FY8G  could  turn  out  to  be  different  from  the  BES  for 
FY85  -  because  the  BES  "didn't  know"  the  correct  funding  figure  for  FY84  and  got 
the  unfunded  carry-over  portion  of  the  FY85  requirement  wrong.  Because  such  late 
decisions  occur,  it  is  true  that  funding  decisions  can  cause  BESs  and  last  looks  to 
differ.  However,  late  funding  decisions  will  normally  not  differ  radically  from  the 
funding  decisions  already  recorded  in  the  BES.  This  means  that  funding  decisions, 
like  program  changes,  are  usually  not  enough  by  themselves  to  explain  large  fluc¬ 
tuations,  like  those  shown  in  Figure  3-1.  (The  turmoil  in  funding  decisions  for  FY86, 
related  to  Congressional  action  on  the  Gramm-Rudman-Hollings  legislation,  serves 
as  a  recent  exception  to  this  rule  and  will  certainly  contribute  to  differences  between 


the  FY87  budget,  submitted  in  the  fall  of  1985,  and  the  last  look  at  FY87,  to  be  taken 
in  the  spring  of  1987.) 

There  are  other  mechanisms  in  the  D041/CSIS/transition  process  that  cause 
estimates  to  change.  These  mechanisms  are  different  in  the  three  phases  of  the 
process.  D041  calculations  are  subject  to  "churn"  in  the  item  data  base,  Stratifica¬ 
tion  treats  a  year  differently  when  it  is  an  outyear  (BY  or  EY)  than  when  it  is  an 
apportionment  year.  And  transition  adjustments  are  not  always  consistent  from  one 
year  to  the  next.  We  shall  look  at  each  of  these  in  turn. 

Figure  3-2  presents  some  preliminary  results  concerning  churn  in  the  D041 
data  base.3  Churn  refers  to  the  tendency  of  item  characteristics  to  change  over  time. 
The  Air  Force  recognizes  that  its  inventory  processes  are  stochastic  -  or 
variable  -  processes,  which  can  only  be  described  in  terms  of  averages  and  other 
statistical  parameters.  In  these  terms,  the  idea  of  churn  is  that  many  of  the 
parameters  the  Air  Force  uses  to  characterize  its  inventory  processes  (e.g.,  demand 
rates,  resupply  times,  prices)  are  not  stable  over  time.  In  other  words,  the  world  is 
not  "steady  state."  Figure  3-2  illustrates  the  potential  effect  of  churn  on  successive 
estimates  of  a  given  year’s  requirement. 

Figure  3-2  shows  that,  in  successive  calculations  of  a  FY84  spares  requirement 
for  F-16As  and  F-lGBs  from  a  September  1982  data  base  and  a  September  1983  data 
base,  approximately  $110.4  million  in  new  requirements  appear  in  the  second 
calculation,  as  a  result  of  changes  in  various  item  characteristics  from  one  data  base 
to  the  next.  (Every  D041  alculation  makes  a  "steady-state”  assumption  for  the  data 
it  uses.)  Both  calculations  were  done  to  achieve  an  80  "ent  aircraft  availability 
rate.  The  second  calculation  assumed  all  the  requirements  Lorn  the  first  calculation 

3For  background  on  churn  und  further  details  on  the  duta  in  Figure  32,  see  Randall  M, 
‘■'ing  and  Virginia  A,  Mattern,  The  Effects  of  Data  Base  Dynamics  in  Estimating  Spares  Costs;  An 
Analysis  of  the  F-16,  Working  Note  AF501-2  (Bethesda,  Maryland:  Logistics  Management 
Institute),  December  1985 


3-7 


were  bought  and  in  the  inventory.  It  is  in  this  sense  that  the  $110.4  million  in 
requirements  are  new —they  reflect  the  failure  of  many  item  factors  to  stay  the  same 
from  one  September  data  base  to  the  next. 

FIGURE  3*2.  D041  DYNAMICS 
(M6A/F-16B  Example) 


In  the  breakdown  of  the  $110.4  million,  roughly  a  third  is  due  to  changes  in 
item  parameters  such  as  failure  rates  per  flying  hour,  repair  times,  and  not- 
reparable-this-station  (NETS)  rates.  Another  third  is  a  result  of  items  that 
presented  nonzero  demand  rates  for  the  first  time.  The  last  third  is  attributable  to 
new  items  -  items  with  stock  numbers  that  were  simply  not  present  in  the  earlier 
data  base.  Very  little  of  the  $110.4  million  is  a  result  of  changes  in  the  planned  oper¬ 
ating  program.  The  underlying  flying-hour  programs  and  fleet  sizes  were  essen¬ 
tially  the  same  for  both  calculations.  Also,  requirements  related  to  replacement  of 
condemnations  and  additives  are  not  included  in  the  $1 10.4  million. 
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As  an  indicator  of  the  relative  size  of  churn  effects,  the  $110.4  million 
represents  approximately  30  percent  of  an  availability-based  estimate  of  $345  mil¬ 
lion  as  the  total  FY84  spares  requirement  (to  achieve  an  80-percent  aircraft 
availability  rate)  for  F-i6As  and  F-16Bs. 

These  results  concerning  churn  are  preliminary.  They  do  not  take  real-world 
asset  levels  into  account,  nor  do  they  reflect  the  fact  that  churn  effects  will  be 
negative  for  some  items,  causing  a  reduction  in  the  overall  effect.  Churn  in  item 
requirements  arising  as  a  result  of  price  changes  (as  opposed  to  changes  in  demand 
rates  and  resupply  times)  was  not  part  of  this  initial  analysis.  Because  only  two  data 
bases  were  examined,  and  then  only  for  components  on  the  F-16A  and  F-16B,  the 
results  cannot  be  generalized  to  the  whole  D041  data  base.  The  results  also  do  not 
take  into  account  that  part  of  the  purpose  of  transition  adjustments  (e.g.,  '’scrubs")  at 
the  end  of  the  D041/CSlS/transition  process  is  to  account  for  churn.  Even  with  these 
qualifications,  however,  there  is  no  question  that  churn  in  the  spares  data  base  is 
another  reason  why  budgets  and  last  looks  will  never  agree  precisely. 

Another  reason  budgets  and  last  looks  tend  to  differ  is  a  result  of  the  way  in 
which  assets  are  treated  in  the  stratification  of  outyear  requirements.  Requirements 
in  earlier  years  are  treated  as  available  assets  in  the  computation  of  later-year 
requirements.  As  a  result,  once  inventory  levels  are  built  in  the  early  years, 
computed  requirements  for  the  outyears  tend  to  look  like  nothing  more  than  the 
replacement  of  condemnations.  As  a  result,  because  budgets  see  a  year  as  the 
extended  year,  and  last  looks  see  it  as  the  apportionment  year,  budgets  tend  to 
understate  in  relation  to  last  looks,  other  things  being  equal.  Again,  transition 
adjustments  are  supposed  to  compensate  for  this  bias,  but  they  are  not  always 
successful  -  particularly  when  the  overall  Air  Force  program  is  in  flux. 

Transition  is  the  third  phase  of  the  D041/CSIS/transition  process.  In  transi¬ 
tion,  the  results  of  item-level  D041/CSIS  computations  are  scrubbed,  errors  are 


corrected,  additive  requirements  not  in  the  item  data  base  are  inserted,  and  other 
management  adjustments  are  made.  Transition  is  so  named  because  it  is  the  process 
that  transforms  a  stratified  requirements  computation  into  an  official  budget.  Lack 
of  consistency  in  transition  adjustments  is  a  third  reason  why  budgets  and  last  looks 
disagree.  Table  3-1  illustrates  an  example  of  this  lack  of  consistency.  It  compares 
transition  adjustments  made  at  Air  Logistics  Centers  for  a  September  1984 
D041/CSIS  computation  with  those  for  a  March  1985  computation  6  months  later.  In 
each  case,  the  adjustments  were  made  to  produce  an  estimate  of  the  FY87  require¬ 
ment.  (FY87  was  the  extended  year  for  both  computations.) 


TABLE  3-1.  INCONSISTENT  TRANSITION  ADJUSTMENTS 

Comparison  of  Transition  Adjustments  for  FY87 
September  1984  Computation  versus  March  1985  Computation 

($000) 


TRANSITION  ADJUSTMENTS 

FY87  (EXTENDED  YEAR) 

September  1984 

March  1985 

Computed  CSIS  deficit 

1,150,610 

1,576,998 

Error  adjustments 

-  106,923 

-  121,213 

Scrub  adjustments 

157,780 

708,911 

Additive  requirements 

205,152 

369,046 

Other  adjustments 

102,037 

16,539 

Total  peacetime  requirement 

1,508,656 

2,550,281 

Table  3-1  illustrates  how  management  decisions  in  transition  can  contribute  to 
differing  estimates  of  the  same  year’s  requirement.  The  bottom  line  in  the  table 
shows  that  even  though  there  is  only  a  6-month  difference  in  age  between  the 
September  1984  D041  data  base  and  the  March  1985  data  base,  the  total  estimated 
requirement  for  FY87  increased  by  more  than  $1  billion.  The  "computed  CSIS 
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deficit"  line  at  the  top  of  the  table  shows  that  about  $426  million  of  this  increase  is 
due  to  changes  within  the  item  data  base  and  in  the  resulting  computed  require 
ment.  Program  changes  and  churn  contribute  to  this  part  of  the  difference.  The 
remainder  of  the  difference,  more  than  $550  million,  is  due  to  changes  in  manage¬ 
ment  decisions  made  about  the  computed  requirement  outside  the  data  base. 
Table  3-1  shows  the  three  categories  of  transition  adjustments  that  account  for  most 
of  the  dollars  in  transition:  error  adjustments,  scrub  adjustments,  and  additive 
requirements.  Other,  smaller  categories  of  transition  adjustments  have  been 
combined  in  the  "other  adjustments"  line. 

The  "error  adjustments"  line  shows  that  the  September  computation  overstated 
the  FY87  requirement  by  $103.9  million  because  of  erroneous  item-level  data  in  the 
data  base.  Errors  in  the  March  data  base  led  to  a  $121.2  million  overstatement. 
Because  6  months  separate  the  data  bases,  it  is  reasonable  to  assume  that  most  of  the 
errors  in  the  September  data  base  were  corrected  in  the  one  for  March.  This  means 
that  most  of  the  $121.2  million  in  errors  in  March  consisted  of  new  (or  newly  dis¬ 
covered)  errors.  Errors  and  error  adjustments  are  a  reflection  of  how  human  error  in 
entering  or  adjusting  data  can  contribute  to  differences  in  estimates. 

The  "scrub  adjustments”  line  shows  that  in  the  6  months  between  September 
1984  and  March  1985,  the  Air  Logistics  Centers  changed  their  statement  of  FY87 
requirements  by  more  than  $550  million.  That  is  the  difference  between  the  scrub 
adjustments  for  September  and  March.  In  this  case,  a  large  portion  of  the  difference 
was  caused  by  a  large  increase  in  additive  requirements  for  the  B-l  bomber, 
requested  by  the  System  Program  Manager  (SPM).  For  the  September  1984  data 
base,  the  SPM  specified  additive  scrub  adjustments  for  the  B-l  of  4- $357.8  million 
for  FY86  and  +$7.6  million  for  FY87.  For  the  March  1985  computation,  these 
estimates  had  increased  to  +$921.9  million  and  +$562.3  million.  In  other  words, 
between  POM  87  (supported  by  the  September  1984  computation)  and  the  budget  for 
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FY87  (supported  by  the  March  1985  computation),  the  SPM  added  more  than 
$800  million  in  B-l  spares  requirements  to  both  the  FY86  requirement  and  the  FY87 
requirement.  This  is  not  to  say  these  new  requirements  were  not  valid.  It  does  show, 
however,  that  Air  Force  efforts  to  bring  budgets  and  last  looks  closer  together  will 
have  to  involve  not  only  better  control  of  the  spares  data  base  but  also  better  control 
and  discipline  in  the  management  decision  process  that  takes  place  outside  the  data 
base. 

The  "additive  requirements"  line  shows  that  estimated  additive  requirements 
(net)  increased  by  more  than  $160  million.  Additive  requirements  are  for  stocks  to 
support  special,  nonrecurring  programs,  such  as  modification  programs  or  spares 
buy-out  programs.  In  transition,  the  adjustment  line  for  additive  requirements  is  for 
additives  outside  the  data  base.  (Additive  requirements  within  the  data  base  are 
adjusted  with  scrub  adjustments.)  Thus,  the  more  than  $160  million  increase  in  the 
additive  adjustment  means  that,  between  September  and  March,  decisions  were 
made  to  do  special  programs  in  FY87  that  were  not  part  of  the  plan  in  September. 

Finally,  not  all  transition  inconsistencies  originate  within  the  logistics  com¬ 
munity.  A  form  of  transition  occurs  between  the  submission  of  an  AFLC  budget 
estimate  to  the  Air  Staff  and  the  Air  Staffs  final  submission  to  OSD.  Air  Staff 
adjustments  to  the  requirement  are  another  reason  why  requirements  estimates 
change.  Although  Air  Staff  adjustments  may  be  necessary  given  the  nature  of  the 
resource  allocation  process  in  the  Pentagon,  they  must,  nevertheless,  be  recognized 
as  contributors  to  the  problem  of  unstable  estimates. 

Many  specific  examples  of  how  management  decisions,  made  after  budgets  have 
been  set,  can  cause  a  given  year’s  requirement  to  change  are  documented  in  the 
CORONA  REQUIRE  study  (cited  earlier),  which  was  conducted  within  the  Air  Force 
in  1983.  The  purpose  was  to  explain  why  and  how  the  last  look  at  the  FY82  spares 
requirement  was  close  to  $1  billion  greater  than  the  budget  for  FY82,  which  at 
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$2,448  billion  (POS)  was  the  largest  spares  budget  the  Air  Force  had  ever  submitted. 

The  following  is  taken  from  the  Executive  Summary  of  the  CORONA 
REQUIRE  report: 

Forty  percent  of  the  final  FY82  requirement  was  generated  by 
additives  or  corrections  resulting  from  off-line  management  decisions,  The 
computer  system  (D041),  commonly  touted  as  the  sole  cause  of  the  problem, 
was  not  the  lone  culprit.  Managers  throughout  the  system  are  allowed  to 
enter  unprogrommed  requirements  into  the  data  base  at  any  time,  As  a 
result,  requirements  grow,  independent  of  funding  considerations. 

The  fact  that  budgets  and  last  looks  have  continued  to  differ  since  the  time  of 
CORONA  REQUIRE  is  evidence  that  many  of  the  problems  identified  are  yet  to  be 
solved. 

Without  arguing  with  the  main  thrust  of  the  CORONA  REQUIRE  conclusion, 
it  is  important  to  note  that  management  decisions  do  not  always  cause  the 
requirement  to  grow  between  budget  and  last  look,  as  CORONA  REQUIRE  implied. 
At  the  time  of  CORONA  REQUIRE,  growth  was  indeed  the  problem.  For  the  years 
FY84  and  FY85,  however,  the  opposite  was  the  case.  As  shown  in  Figure  3-1,  the 
last  looks  for  FY84  and  FY85  were  well  under  the  original  budget  estimates.  Some 
of  this  change  was  due  to  changes  in  the  data  base,  and  some  to  management 
decisions  and  estimates  that  the  requirement  would  be  smaller.  In  this  period,  for 
example,  Air  Force  management  was  arguing  that  cost  savings  resulting  from 
increases  in  competition  were  bringing  spares  requirements  down.4 

The  fact  that  last  looks  were  smaller  than  the  budget  estimates  for  FY84  and 
FY85  made  no  difference  in  the  credibility  problem.  In  fact,  Congress  questioned  the 
validity  of  requirements  estimates  for  FY86  and  FY87  partly  on  the  grounds  that  the 


*An  alternative  approach  would  have  beer  to  say  that  cost  savings  from  competition 
initiatives  were  allowing  the  Air  Force  to  buy  better  supply  support  and  more  readiness  from  the 
approved  budget.  Another  possible  approach  would  have  been  some  compromise  between  the  two; 
some  reduction  in  requirements,  along  with  some  improvement  in  support. 
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Air  Force  was  not  able  to  spend  all  it  had  asked  for  and  gotten  in  its  FY84  and  FY85 
spares  budgets. 

Finally,  changing  item  prices  — and  the  many  factors  that  can  cause  prices  to 
change— are  another  potentially  significant  reason  why  requirements  estimates 
may  change  from  budget  to  last  look.  Competition  advocacy  and  breakout  programs, 
changes  in  contracting  and  ordering  procedures,  and  re-pricing  agreements  all  will 
cause  estimates  to  change  -  sometimes  significantly. 

Taken  together,  the  many  reasons  and  pressures  discussed  in  this  chapter 
explain  how  budget  and  last- look  estimates  can  differ  by  amounts  as  large  as  those 
in  Figure  3-1.  The  next  question  is  whether  forecasting  tools,  like  ALERT,  can  some¬ 
how  rise  above  the  behavior  of  the  budgeting  and  execution  system  and  make 
accurate  and  stable  forecasts  of  spares  requirements  for  POMs.  If  not  -  and  the  next 
chapter  argues  that  ALERT  cannot  -  the  Air  Force  has  no  alternative  but  to  instill 
greater  discipline,  control,  and  stability  in  the  programming,  budgeting  and 
execution  process,  if  the  credibility  problem  for  spares  requirements  is  to  be  solved. 
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4.  HOW  ALERT  WORKS  AND  WHY  IT  WON’T  SOLVE 
THE  CREDIBILITY  PROBLEM 


FOUR  CRITERIA  FOR  EVALUATING  A  POM  FORECASTING  MODEL 

The  Air  Logistics  Early  Requirements  Technique  (ALERT)  forecasts  aircraft 
POS  spares  requirements  for  POMs.  It  does  so  by  attempting  to  predict  what  the 
"last-look”  estimate  with  the  D041/CSIS/transition  system  will  be  for  each  of  the 
5  POM  years.  For  example,  ALERT’s  POM  88  forecasts  are  projections  of  what  the 
last  looks  will  be  for  the  years  FY88,  FY89,  FY90,  FY91,  and  FY92. 

ALERT  forecasts  are  produced  with  equations  derived  through  regression  anal¬ 
ysis  of  historical  spares  requirements.  Separate  linear  regressions  are  done  for  each 
of  17  Air  Force  "systems”  that  generate  aircraft  spares  requirements,  including: 
14  different  aircraft  types  (by  M/D  -  Mission/Design),  the  F-100  engine,  common 
components,  and  "other  systems.”  Historical  requirements  data,  budget  projections, 
age-of-the-fleet  and  value-of-the-fleet  data,  and  certain  other  technical  data  serve  as 
data  points  and  inputs  to  the  regressions.  Every  year,  with  the  addition  of  another 
year’s  worth  of  history,  ALERT  regressions  are  redone  and  new  equations  are 
developed  for  the  POM  forecasts  to  be  made  that  year. 

ALERT’s  success  or  failure  as  a  forecasting  model  will  be  determined  by  four 
things:  its  conceptual  validity,  its  degree  of  precision,  its  ability  to  deliver  stable 
estimates,  and  its  predictive  accuracy.  Conceptual  validity  and  degree  of  precision 
are  internal  characteristics,  determined  by  the  way  the  model  works  and  the  input 
data  it  has  to  work  with.  Stability  and  accuracy  are  external  characteristics, 
determined  by  the  behavior  of  the  model’s  forecasts  over  time  and  the  level  of  accu¬ 
racy  they  achieve.  Of  the  four  characteristics,  stability  and  accuracy  of  forecasts  are 

the  most  important.  If  a  model  cannot  d*  liver  stable  and  accurate  forecasts,  it  will 

\ 

not  solve  the  credibility  problem  for  spares. 


4  1 


V| 

1  ( 

■A 


s 


f'cl 

t, 


ft 

V', 


a 


ALEET’s  characteristics  and  performance  in  each  of  the  four  areas  are  eval¬ 
uated  in  this  chapter.  As  we  shall  see,  ALERT  has  problems  in  every  area.  The  most 
serious  are  in  the  stability  and  accuracy  of  its  forecasts.  All  of  the  problems 
discussed  are  due,  either  directly  or  indirectly,  to  the  volatility  of  D041/CSIS  esti¬ 
mates,  which  play  a  central  role  in  the  ALERT  methodology.  The  conclusion  is  that 
ALERT  cannot  and  will  not  solve  the  credibility  problem  for  spares.  We  begin  with  a 
review  of  how  the  model  works. 

HOW  ALERT  WORKS 


ALERT  was  developed  by  the  Materiel  Management  Directorate  at  AFLC 
(AFLC/MMMA)  in  the  fall  of  1983.  The  ALERT  regression  approach  extends  an 
approach  employed  in  another  POM  forecasting  model  for  spares,  the  POS  Spares 
Estimating  Model  (POSSEM),  which  was  developed  by  the  Air  Staff  (HQ  USAF/AC). 
ALERT  was  used  officially  for  the  first  time  for  POM  87.  POSSEM  was  used  for 
POM  86  and  POM  85.  Prior  POMs  were  done  with  a  cost-per-flying-hour  (CPFH) 
method.  To  obtain  POM  spares  estimates,  the  CPFH  method  applies  a  CPFH  factor, 
derived  from  the  budget-year  requirement,  to  the  planned  flying-hour  program  for 
the  POM  years.  The  CPFH  method  is  flawed  by  its  failure  to  recognize  spares 
requirements  as  net  requirements,  POSSEM  and  ALERT  were  intended  to  replace 
the  CPFH  method  and  serve  as  improved  POM  forecasting  techniques. 

In  describing  the  ALERT  methodology,  we  shall  use  POM  88  am  an  example. 
The  description  may  be  applied  to  any  other  POM  cycle  by  adding  (or  subtracting) 
the  same  number  of  years,  as  appropriate,  to  all  numerical  expressions  for  years  that 
appear  in  the  description. 

AFLC  must  submit  POM  88  estimates  to  the  Air  Staff  in  the  fall  of  1985, 
2  years  before  the  first  POM  year,  FY88.  The  unscrubbed  estimates  from  the 
June  1985  D041/CSIS  computation  are  the  most  current  D041/CSIS  estimates  avail¬ 
able  in  time  to  meet  the  fall  deadline.  The  June  1985  estimates  are  for  the  current 


year  (CY  -  the  last  quarter  of  FY85),  the  apportionment  year  (AY  -  FY86),  the 
budget  year  (BY  -  FY87),  and  the  extended  year  (EY  -  FY88). 

The  variable  to  be  predicted  in  an  ALERT  forecast  is  a  D041/CSIS/transition 
last  look.  For  POM  88,  ALERT  attempts  to  predict  the  last  look  for  each  of  the  years 
FY88  through  FY92.  The  historical  data  underlying  the  forecast  are  actual  last-look 
values  for  FY78  through  FY85.  Because  it  is  the  fall  of  1985,  FY85  is  the  last  year 
for  which  an  actual  last  look  is  available  (from  the  March  1985  D041/CSIS/transition 
computation).  There  is  no  history  yet  for  the  2  remaining  years,  FY86  and  FY87, 
that  lie  between  FY85  and  the  first  POM  year,  FY88.  Instead,  ALERT  uses 
scrubbed  estimates,  again  from  the  March  1985  computation,  as  the  "history"  for 
those  2  years. 

This  last  point  is  important.  It  identifies  the  first  instance  in  which  ALERT 
relies  upon  an  early  look  with  the  D041/CSIS/transition  system  to  say  what  a  last 
look  will  be.  A  last  look  is  the  last  in  the  sequence  of  six  D041/CSIS/transition  runs 
that  see  a  given  year.l  The  March  1985  D041/CSIS/transition  computation  repre¬ 
sents  only  the  fourth  look  at  FY86  and  only  the  second  look  at  FY87.  In  each  case, 
however,  ALERT  treats  these  estimates  as  if  they  were  last  looks.  Of  course,  ALERT 
does  not  have  much  choice.  Something  must  be  used  as  history  for  FY86  and 
FY87  -  trying  to  forecast  for  POM  88  on  the  basis  of  history  that  extends  through 
FY85  only  is  not  very  palatable  -  and  it  is  hard  to  imagine  other  variables  that 
could  be  counted  on  to  do  any  better.  Nevertheless,  because  of  volatility  in  the 
D041/CSIS/transition  system,  we  will  see  that  use  of  early  looks  as  history  seriously 
diminishes  ALERT’S  ability  to  make  accurate  forecasts. 


'Beginning  with  a  June  computation,  the  D041/CSIS  system  (which  is  run  quarterly)  will 
compute  a  given  year's  requirement  a  total  of  13  times  Of  those  13  computations,  only  6  undergo 
transition  (scrubs)  These  6  D041/CSIS  runs  with  full  transition  are  the  6  "looks"  at  a  given  year 
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Figure  4-1  shows  the  historical  values  of  the  dependent  variable  (last  looks)  for 
one  weapon  system,  the  C-135  aircraft,  that  went  into  the  ALERT  regression  anal¬ 
ysis  for  POM  88.  For  its  POM  88  forecasts,  ALERT  used  historical  data  for  the  years 
FY78  through  FY85.  As  explained  above,  the  values  shown  for  FY86  and  FY87  are 
estimates,  not  true  le  st  looks. 


ALERT  is  a  stepwise,  multilinear  regression  model.  As  a  regression  model,  it 
approaches  the  problem  of  projecting  the  next  five  values  (for  the  years  FY88 
through  FY92)  for  a  system  like  the  C-135  by  asking:  Is  there  a  set  of  observable 
variables  that  "explains”  most  of  the  variation  in  the  system’s  requirements? 

If  we  can  find  such  variables,  as  well  as  a  functional  relationship  between  them 
and  the  last-look  variable  we  are  trying  to  project,  we  can  use  that  relationship  to 


forecast  future  last  looks.  Regression  is  a  standard  statistical  method  for  finding 
such  relationships.^ 

The  independent  predictor  variables  used  in  ALERT  regressions  are:  the  dollar 
value  of  the  fleet  by  system  (for  the  17  systems  addressed  by  ALERT);  the  reciprocal 
of  the  age  of  the  fleet  by  system;  a  dummy  variable  to  control  for  what  AFLC  deems 
are  nonrecurring  phenomena  in  the  historical  data;  and  the  computed  (but 
unscrubbed)  CSIS  deficit  for  the  extended  year  from  the  June  D041/CSIS  compu¬ 
tation  preceding  ALERT  projections  each  fall.  The  value  of  the  computed  D041/CSIS 
deficit  is  modified  by  addition  of  the  value  of  on-order  assets  at  the  beginning  of  the 
extended  year  as  calculated  in  the  June  computation. 3  ALERT  also  uses  calendar 
year  occasionally  as  an  independent  predictor  variable. 

With  the  exception  of  the  dummy  variable,  which  serves  as  a  mechanism  for 
adjusting  ALERT  forecasts  based  on  AFLC  management  inputs,  each  of  ALERT's 
independent  variables  can  -  when  necessary  -  be  observed,  measured,  or  predicted. 
What  the  age  and  value  of  the  fleet  will  be  at  the  time  of  a  last  look  can  be  predicted 

*ALF.RT  is  not  a  time-aeries  regression.  In  a  time-series  analysis,  the  chronologicul  order  in 
which  historical  last  looks  present  themselves  (e.g,,  the  order  in  Figure  4-1)  would  influence  the 
projected  values  of  future  last  looks.  This  is  not  the  case  for  ALERT  projections.  In  the  case  of  the 
C-135,  for  example,  changing  the  position  of  the  last  looks  over  the  horizontal  axis  (the  time  axis)  in 
Figure  4  1  would  not  chunge  the  forecasting  equations  and  finul  projections  that  ALERT  derives 
from  those  data.  Reuders  are  cautioned,  therefore,  to  be  careful  when  they  think  of  ALERT  as 
"fitting  a  straight  line"  to  the  data  shown  in  Figure  4-1.  ALERT  finds  a  linear  relationship  between 
those  duta  und  other  variables,  but  it  does  not  attempt  to  say  thut  requirements  are  a  linear  func¬ 
tion  of  time.  (An  exception  is  ALERT's  occasional  use  of  the  calendar  year  us  un  Independent, 
predictor  variable  -  discussed  later  ) 

3The  reason  given  in  ALERT  documentation  for  adding  the  value  of  on-order  assets  to  tho 
computed  deficit  is  that: 

Inspection  of  past  CSIS  products  indicated  the  "total  buy"  (the  sum  of  on-order  plus 
the  deficit)  to  be  more  closely  aligned  with  eventual  (last  look)  requirements,  The 
computed  deficit  appeared  to  be  understated  relative  to  the  eventual  budget  (the  last 
look)  throughout  the  history  of  the  data  Reusons  were  not  anul.vzed 

We  shall  return  to  this  point  in  the  discussion  of  ALERT’s  conceptual  validity,  For  now,  one  of  the 
reasons  early  CSIS  deficits  are  consistently  understated  relative  to  last  looks  is  that  stratification 
converts  near-year  requirements  into  outyear  assets.  This  makes  outyear  requirements  look  like 
nothing  more  than  the  replacement  of  condemnations.  As  we  have  seen  in  Chapter  3,  however, 
reparables  requirements  Involve  substantially  more  thun  replacement  of  condemnations  each  year 


with  reasonable  certainty  from  force  structure  plans.  The  value  of  the  D041/CSIS 
variable  is  directly  observable  from  the  June  computation  that  precedes  the  ALERT 
forecast. 

Still  using  POM  88  as  an  example,  ALERT’s  stepwise  regression  proceeds  as 
follows:  For  the  first  POM  year,  FY88,  ALERT  regresses  historical  values  of  the 
June  D041/CSIS  variable  against  actual,  historical  last-look  values  for  FY78 
through  FY85  (and  the  estimated  history  for  FY86  and  FY87).  The  June  D041/CSIS 
variable  is  always  a  predictor  variable  for  the  first  POM  year.  For  the  regression, 
historical  values  of  the  independent  D041/CSIS  variable  are  appropriately  time- 
lagged  against  historical  last  looks.  For  example,  the  value  of  the  June  1981  D041/ 
CSIS  computation  is  associated  with  the  last  look  for  FY84,  The  regression  is 
stepwise  in  that  it  may  choose  one  additional  variable  from  the  remaining  indepen¬ 
dent  variables  (age,  value,  or  year)  as  an  additional  predictor,  if  the  addition  of  that 
variable  will  '’enhance"  ALERT's  predictive  ability.  The  degree  of  "enhancement"  is 
judged  by  the  coefficient  of  determination  (r2)  obtained  in  the  regression, 

For  the  POM  outyears,  the  second  through  fifth  POM  years,  there  is  no  D041/ 
CSIS  variable  available  for  use  as  an  independent  variable  in  a  regression.  There¬ 
fore,  ALERT  regresses  against  value  of  the  fleet,  enhanced  by  stepwise  selection  of 
one  more  of  the  remaining  variables  -  either  age  of  the  fleet  or  calendar  year. 

D041/CSIS  products  still  influence  the  outyear  projections,  however.  Just  as 
ALERT  uses  D041/CSIS  early-look  estimates  as  history  for  the  2  years  preceding  the 
first  POM  year,  ALERT  treats  its  own  projection  for  the  first  POM  year  as  1  more 
year  of  "history"  for  the  outyear  regressions.  Thus,  D041/CSIS  system  estimates  of 
near-year  requirements  indirectly  influence  outyear  projections. 

The  dummy  independent  variable  in  ALERT  assumes  the  value  zero  or  one  for 
the  regression.  The  value  of  one  is  used  for  any  historical  years  (or  future  years)  that 
AFLC  management  believes  had  (or  will  have)  abnormally  high  requirements. 
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The  ALERT  regressions  result  in  equations  that  express  the  dependent 
variable  (last  look)  as  a  linear  combination  of  the  independent  variables.  These 
equations  are  the  ones  used  to  make  POM  forecasts.  The  first  two  equations  in 
Figure  4-2  are  generic  examples.  The  first  equation,  for  the  first  POM  year,  always 
involves  at  least  the  June  D041/CSIS  variable  and  perhaps  one  of  the  enhancer 
variables  (ENH),  a  constant  term,  and  the  dummy  variable  called  PEAK,  to  control 
for  abnormal  peaks  in  the  requirement.  The  second  equation  in  Figure  4-2  shows  the 
form  of  the  forecasting  equations  for  the  POM  outyears.  The  coefficients  Bi  and  Ci 
are  derived  in  the  ALERT  regressions. 

FIGURE  4*2.  ALERT  FORECASTING  EQUATIONS 


1ST  ROM  YEAR  ■  Bo  +  Bi(D041/C5IS)  ♦  B2(INH)  +  Bj(PIAK) 

2ND  -  STH  POM  YEARS  ■  C0  +  Ci(VALUl)  +  C2(INH)  ♦  Cj(PEAK) 

E.G., 

C*1 33  FYSB  REQUIREMENT  ■  383.7  ♦  (  -  170)(D041)  ♦  ( -  6231  )(1/AQE) 
$63.5  MILLION  .  383.7  +  ( -  170)(0.564)  ♦  ( -6231  HO. 036) 


ALERT  produces  such  equations  for  all  of  its  17  systems.  The  equation  at  the 
bottom  of  Figure  4-2  is  the  equation  ALERT  produced  to  forecast  FY88  requirements 
for  the  C-135  for  POM  88.  The  computed  deficit  in  FY88  for  the  C-135  (from  the 
June  1985  D041/CSIS  computation,  with  the  value  of  on-order  assets  added)  is 
$0,564  million,  The  average  age  of  the  C-135  fleet  in  FY88  will  be  27.8  years,  with  a 
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reciprocal  value  of  0.036.  The  predicted  value  of  the  last  look  for  C-135  requirements 
inFY88  is  $63.5  million.4 

WHY  ALERT  WON’T  SOLVE  THE  CREDIBILITY  PROBLEM 
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ALERT  will  not  solve  the  credibility  problem  for  spares  because  it  has  concep¬ 
tual  problems,  precision  problems,  accuracy  problems,  and  stability  problems. 

Conceptual  Problems 

Conceptually,  ALERT  has  many  appealing  features.  Regression  is  a  stan¬ 
dard  forecasting  technique,  and  ALERT  is  innovative  in  its  use  of  early  look  D041/ 
CSIS/transition  estimates  as  inputs  for  projections  of  what  last  looks  will  be. 5 
Value  and  age  of  the  fleet  make  sense  (and  have  been  used  elsewhere  in  DoD)  as 
predictors  of  spares  requirements.  And,  from  a  practical  perspective,  the  use  of  a 
dummy  variable  to  allow  input  from  AFLC  management  is  a  good  idea.  AFLC  is  the 
Air  Furce  organization  responsible  for  spares  management  and  the  most 
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4The  equations  for  the  C-135  shown  in  Figure  4-2  are  direct  ALERT  output  produced  in 
September  of  1985  In  general,  the  technical  evaluation  of  ALERT  in  this  chapter  is  based  on  direct 
model  outputs,  which  may  not  reflect  final  management  adjustments.  (Before  ALERT  forecasts 
become  official,  they  are  subject  to  management  adjustment,  both  by  AFLC  und  by  the  Air  Staff.) 
As  a  result,  the  final  C-135  FY88  projection  delivered  to  the  Air  Staff  in  the  fall  of  1985  may  differ 
from  the  projection  in  Figure  4-2. 

5[n  developing  a  regression  model,  it  is  good  practice  to  choose  independent  variables  that  con 
be  related  in  some  sensible  way  to  the  dependent  variables,  It  is  very  reasonable  in  concept,  there¬ 
fore,  to  choose  early  D041/CSIS/transition  estimates  as  the  key  indicator  of  what  later  D041/CSIS/ 
transition  last  looks  will  be.  Recognizing  that  early  looks  and  last  looks  do  differ,  it  is  also  appeal¬ 
ing  that  ALERT  uses  the  early  estimates  as  inputs  to  a  regression  analysis,  ruther  than  attempting 
to  use  them  directly.  This  indirect  use  of  early  D041/CSIS  estimates  was  deliberate  in  the  course  of 
ALERT’s  development.  The  following  statement  from  ALERT  documentation  describes  a  key 
assumption  underlying  the  ALERT  approach: 

The  summary  of  the  DQ41  item-by  item  computation  of  the  buy  requirement  pro¬ 
vides  valuable  information  relating  to  the  final  requirement  at  the  Vl/D  level,  and 
represents  the  most  likely  predictor  for  eventual  requirements.  Due  to  the 
dynamics  over  time,  this  computation  is  subject  to  change  One  of  the  objectives  of 
the  ALERT  model  is  to  capture  the  direction  and  strength  of  this  change 

We  cannot  argue  with  this  concept,  but  the  central  finding  of  this  study  (reflected  in  the 
previous  chapter  and  in  later  analysis  of  ALERT’s  stability  and  accuracy)  is  that  there  is  so  much 
volatility  in  the  D041/CSIS/transition  system  that,  even  though  it  may  be  the  best  predictor  concep¬ 
tually,  it  proves  a  very  bad  predictor  in  practice. 
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knowledgeable  about  spares  requirements.  It  makes  sense  to  provide  a  mechanism 
for  incorporating  AFLC  judgment  into  the  spares  forecasting  model  used  for  POMs. 

But  ALERT  does  have  some  conceptual  problems  that  reduce  its  credi¬ 
bility.  For  example,  the  fact  that  new  equations  are  developed  and  used  to  forecast 
each  year  automatically  undermines  the  credibility  of  previous  forecasts  made 
earlier.  This  is  particularly  a  problem  when  a  given  year  becomes  a  first  POM  year, 
after  having  been  a  POM  outyear  for  4  years.  For  the  first  POM  year,  the 
introduction  of  the  D041/CSIS  variable  into  the  predicting  equation  makes  it  highly 
unlikely  that  the  new  prediction  will  be  consistent  with  earlier  predictions.  (We 
shall  see  evidence  of  this  later  in  the  discussion  of  ALERT’s  stability  problems.) 

Another  conceptual  problem  is  in  ALERT’s  occasional  use  of  the  calendar 
year  as  a  predictor  variable.  Given  the  simple  linear  form  of  ALERT’s  forecasting 
equations,  use  of  the  calendar  year  as  a  predictor  variable  implies  that  the  spares 
requirement  for  a  system  will  change  by  the  same  amount  each  year.  The  fact  that 
th's  may  have  been  the  case  over  some  sequence  of.years  in  the  past  is  no  reason  to 
believe  it  will  continue  into  the  future.  Spares  requirements  are  not  necessarily 
time-independent  in  all  respects,  but  a  simple  linear  relationship  is  unlikely. 

Another  of  ALERT’s  conceptual  problems  is  the  combination  of  assets  and 
requirements  in  the  D041/CSIS  predictor  variable.  Assets  and  requirements  affect 
the  requirement  in  opposite  ways.  Combining  assets  and  requirements,  therefore, 
yields  a  hybrid  variable  that  has  no  intuitive  content  or  sensible  relationship  to  the 
predicted  variable.  Without  such  a  relationship,  regressions  tend  to  be  artificial  and 
unconvincing. 

ALERT’s  most  serious  conceptual  problem,  in  fact,  is  that  the  predicting 
equations  it  generates  cannot  be  understood  in  terms  of  any  reasonable  physical 
relationship.  As  a  result,  there  is  no  conceptual  reason  to  believe  they  are  good 
predictors.  To  illustrate  the  equations  below  are  the  FY88  forecasting  equations 
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ALERT  produced  for  the  F-16  and  F-15  for  POM  88.  For  both  systems,  the  D041/ 
CSIS  variable  is  the  only  predictor  variable: 

F-16  FY88  requirements  =  73.3  +  (  r  66.9)  (D041/CSIS) 

F-15  FY88  requirements  =  139.0  +  ( -42.0)  (D041/CSIS) 

The  equations  are  similar  except  that  the  coefficients  for  the  D041/CSIS 
variables  have  opposite  signs.  In  the  case  of  the  F-16,  the  positive  sign  implies  that 
the  greater  an  earlier  estimate  of  FY88  requirements,  the  greater  those  require¬ 
ments  will  eventually  be.  This  would  seem  to  make  sense.  But  then  for  the  F-15,  the 
opposite  is  true:  The  greater  the  earlier  estimates,  the  smaller  the  eventual  FY88 
requirements.  The  use  of  the  D041/CSIS  variable  as  a  predictor  of  spares 
requirements  only  makes  sense  if  it  works  the  same  way  for  all  systems.  In  general, 
the  fact  that  signs  vary  U  an  indicator  within  ALERT  itself,  that  early  D041/CSIS 
estimates  are  highly  unreliable  as  predictors  of  eventual  requirements. 

Precision  Problems 

A  model’s  precision  is  different  from  its  accuracy,  in  the  same  way  that 
the  scatter  pattern  of  a  shotgun  is  different  from  the  marksmanship  of  the  shooter. 
Here,  we  look  at  ALERT’s  internal  precision  -  its  scatter  pattern. 

Regression  relationships  are  relationships  between  mean  or  average 
values  of  the  independent  and  dependent  variables.  They  can  be  counted  on  to  be 
exactly  accurate  for  only  the  sample  means  of  the  observed  values  of  those  variables. 
Away  from  those  means,  regression  predictions  are  nothing  more  than  estimates  of 
the  mean  value  of  a  random  variable.  When  ALERT  makes  a  prediction,  therefore, 
it  is  estimating  the  mean  value  of  a  variable  that  can  take  on  many  different  values. 
ALERT’s  accuracy  is  a  question  of  whether  it  is  a  good  predictor  of  the  mean  —  is  the 
shooter  a  good  marksman?  Its  precision  is  a  measure  of  the  inexactness  of  the 
estimate  -  how  big  is  the  scatter  pattern?  This  inexactness  is  a  function  of  sampling 
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error  in  the  estimate  of  the  mean  and  of  variance  in  the  dependent  (predicted) 
variable  that  is  not  accounted  for  by  the  regression  relation.  The  amount  of 
inexactness  in  a  prediction  can  be  quantified  in  terms  of  prediction  intervals. 
Prediction  intervals  are  interval  estimates  of  future  values  of  the  dependent  vari¬ 
able,  centered  on  point  estimates  of  the  mean.  Associated  with  every  prediction 
interval  is  a  probability  that  the  projected  interval  has  "captured”  the  desired  value 
of  the  dependent  variable.  This  probability  and  the  width  of  the  interval  provide  a 
measure  of  the  inexactness  of  the  prediction.  Figure  4-3  illustrates  a  prediction 
interval  around  ALERT'S  point  estimate  of  $55.2  million  as  the  C-5's  FY88 
requirement. 


FIGURE  4-3.  ALERTS  STATISTICAL  ERROR  AT  THE  SYSTEM  LEVEL 


MILLIONS 


FISCAL  VIAK 


Figure  4-3  expresses  the  ALERT  projection  for  the  C-6  in  the  following 
way:  Given  the  historical  values  of  last  looks  for  the  C-5  from  FY78  through  FY87 
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(plotted  in  the  figure),  and  the  way  the  ALERT  works,  there  is  a  30.2  percent  chance 
that  the  interval  defined  by  $55.2  million  plus  or  minus  50  percent  of  itself  (i.e.,  the 
interval  from  $27.6  million  to  $82.8  million)  includes  the  C~5’s  FY88  requirement. 
Put  another  way,  we  can  say  with  only  30.2  percent  confidence  that  the  C-5’s  FY88 
requirement  will  be  between  $i,7.6  million  and  $82.8  million.  In  other  words,  the 
point  estimate  of  $55.2  million  is  not  very  precise. 

In  fact,  ALERT  developers  have  acknowledged  that  ALERT  does  not 
generate  reliable  estimates  at  the  weapon-system  level.  The  data  in  Table  4-1  are 
taken  from  an  AFLC  comparison  of  ALERT  with  the  POSSEM  model.  For  each 
weapon  system,  AFLC  tested  ALERT’s  ability  to  interpolate  historical  requirements 
levels,  given  information  about  true  last  looks  on  both  sides  of  the  interpolated  year. 
This  was  done  for  each  year  from  FY78  through  FY86.  For  each  year  and  each 
weapon  system,  AFLC  measured  the  mean  absolute  percentage  error  (MAPE) 
between  the  ALERT  interpolation  and  the  actual  last  look  for  the  year.  These 
MAPEs  were  then  weighted  by  the  size  of  the  requirement  in  each  year  and 
averaged,  to  yield  a  single  weighted,  weapon-system  MAPE  for  the  period  FY78 
through  FY86.  As  the  table  shows,  ALERT’s  percentage  error  ranged  from  a  low  of 
23  percent  to  a  high  of  112  percent,  with  an  (unweighted)  overall  average  error  of 
63  percent  per  system. 

Although  precision  at  the  system  level  is  desirable,  it  is  not  crucial  for 
POMs.  For  POMs,  the  aggregate  estimate  is  of  prime  importance.  What  is  ALERT’s 
precision  at  the  aggregate  level? 

Summing  across  all  17  of  its  systems,  ALERT’s  precision  is  better  in  the 
aggregate  than  it  is  at  the  system  level.  For  example,  an  interval  estimate  of 
$2.15  billion,  plus  or  minus  $430  million  (20  percent),  has  an  87-percent  chance  of 
"capturing”  the  aggregate  FY88  requirement.  This  assumes  that  the  point  estimate 
of  $2.15  billion  is  an  accurate  estimate  of  the  mean.  A  tighter  prediction  interval  of 
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TABLE  4-1.  ALERT  IMPRECISION  AT  THE 
WEAPON-SYSTEM  LEVEL:  FY78  -  FY86 


M/D 

PERCENT 

ERROR 

M/D 

PERCENT 

ERROR 

A-7 

112 

F-4 

23 

A- 10 

85 

F-5 

109 

B-52 

85 

F- 1 5 

50 

C-130 

64 

F-16 

35 

C-135 

66 

F-111 

47 

C-141 

78 

F-100 

29 

c-s 

63 

Other 

24 

E-3 

47 

T-38 

91 

Mott.*  Unwtighttd  »v*r*gt:  63  ptretnt 


$2.15  billion  plus  or  minus  $215  million  (10  percent)  has  a  55-percent  chance  of 
covering  the  requirement.  How  these  aggregate  prediction  intervals  were  computed 
is  explained  in  the  Appendix. 

Accuracy  Problems 

The  estimate  of  the  aggregate  FY88  POS  requirement,  the  sum  of 
ALERT’s  17  system-level  estimates  for  POM  88,  is  $2.15  billion.  We  have  just  seen 
that  if  this  is  an  accurate  estimate  of  the  mean  aggregate  requirement,  we  can  be 
fairly  confident  that  the  actual  FY88  requirement  will  be  within  $430  million  of  that 
figure.  But  why  should  we  believe  that  $2.15  billion  is  an  accurate  estimate?  So  far, 
all  we  have  is  the  concept  of  ALERT,  namely,  that  regression  can  give  us  the  right 
answer,  using  early  D041/CSIS  estimates  and  information  about  age  and  value  of 
the  fleet  as  inputs.  Is  there  any  external,  objective  way  to  test  ALERT’s  accuracy? 

Because  POM  87  was  the  first  POM  done  officially  with  ALERT,  the  first 
official  test  of  ALERT’s  predictive  accuracy  will  not  come  until  the  spring  of  1987, 
when  the  results  of  the  March  1987  last  look  will  become  available.  In  the 


meantime,  we  can  test  the  model’s  potential  accuracy  by  applying  it  to  "project”  a 
known  historical  value.  In  particular,  if  we  apply  the  ALERT  regression 
methodology  to  the  appropriate  historical  data  for  FY78  through  FY84,  we  can  see 
how  well  ALERT  does  in  projecting  the  last  look  at  FY85.  Figure  4-4  shows  the 
results. 

FIGURE  4-4.  ALERT*!  PREDICTIVE  ACCURACY 


! 


Using  the  same  predictor  variables  by  system  as  in  ALERT’s  POM  88 


equations,  but  using  the  data  that  would  have  been  available  at  the  time  (late  in 
FY82,  when  POM  85  projections  were  being  made),  we  performed  regressions  and 
obtained  ALERT-type  projections  of  what  the  last  look  FY85  would  be.  As 


Figure 4-4  shows,  ALERT’s  projection  was  $3.8  billion.  The  actual  FY85  last  look 


L 


4-14 


(for  the  17  systems  considered  by  ALERT)  was  $2.3  billion.®  ALERT  was  wrong  by 
$1.5  billion  -  an  error  of  65  percent. 

How  can  the  model  have  been  this  wrong?  To  answer  this  question,  we 
repeated  the  experiment,  but  with  a  crucial  change  in  the  input  data.  The  projection 
just  described  used  input  data  that  would  have  been  available  to  ALERT  at  the  time 
(fall  of  1982),  when  a  projection  for  FY85  was  being  made.  In  particular,  as  the 
history  for  FY84,  we  used  an  estimate  from  the  D041/CSIS/transition  system  of  what 
the  last  look  would  be.  In  the  fall  of  1982,  ALERT  has  only  a  second-look  estimate 
for  FY84.7  The  actual  last  look  for  FY84,  in  aggregate  for  the  17  systems  in  ALERT, 
was  $2.5  billion,  In  the  fall  of  1982,  however,  the  D041/CSIS/transition  system, 
taking  its  second  look  at  FY84,  was  estimating  the  FY84  requirement  to  be 
$3.3  billion.  When  we  replaced  that  estimate  with  the  actual,  historical  last  look  for 
FY84,  however,  the  ALERT  procedure  yielded  an  estimate  of  $2.54  billion  as  the 
aggregate  requirement  for  FY85.  This  projection  is  within  $250  million  (11  percent) 
of  the  actual  FY85  last  look  of  $2.3  billion. 

There  are  negative  and  positive  aspects  to  this  result.  On  the  negative 
side,  the  result  says  that  ALERT’s  reliance  on  the  D041/CSIS/transition  system  for 
the  last  2  years  of  history  causes  the  method  to  be  substantially  less  accurate  than  it 
would  be  otherwise.  Unfortunately,  the  data  ALERT  needs  to  be  accurate  -  actual 
last  looks  for  the  2  years  preceding  the  first  POM  year  -  cannot  be  known  with 
certainty  at  the  time  ALERT  must  make  its  projections.  On  the  positive  side,  it  says 
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0The  total  last  look  at  FY85  of  82 ,6  billion  Included  $300  million  for  the  B-l,  which  Is  not 
included  among  the  17  systems  addressed  by  ALERT,  Thus,  $2,3  billion  Is  the  correct  portion  of  the 
FY85  last  look  to  compare  with  the  ALERT  projection  of  $3,8  billion. 


‘dn  both  tests,  we  used  the  actual,  historical  last  look  for  FY83  us  the  history  for  FY83 
Strictly  speaking,  wo  should  have  used  a  fourth-look  estimute  for  FY83,  because  that  is  all  thut 
would  have  been  available  in  the  full  of  1982,  In  fact,  there  was  less  than  a  $100  million  difference 
between  the  fourth  and  last  looks  at  FY83,  The  last  look  at  FY83  was  $3,28  billion,  The  fourth  look 
was  $3.19  billion. 
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that  if  the  budgeting  system  were  improved,  the  accuracy  of  ALERT  forecasts  for 
POMs  would  probably  improve  as  well. 

Stability  Problems 

It  is  not  likely  that  ALERT  forecasts  of  a  given  year’s  requirement  will  be 
stable  over  time.  Successive  ALERT  forecasts  of  the  FY88  requirement,  for  example, 
have  been  $5.0  billion  (spring  1984),  $3.44  billion  (spring  1985),  and  $2.15  billion 
(fall  1985).  The  $5.0  billion  estimate  for  FY88  appears  in  a  1984  description  of 
ALERT  prepared  for  the  Deputy  Chief  of  Staff/Materiel  Management  at  AFLC.  The 
$3.44  billion  estimate  appears  in  an  April  1985  memorandum  from  the  Comptroller, 
HQ  USAF,  to  the  Comptroller,  HQ  AFLC.  The  $2.15  billion  estimate  is  the  sum  of 
17  system-level  ALERT  estimates  taken  directly  from  ALERT  output  products  pre¬ 
pared  in  the  fall  of  1985, 

Because  these  estimates  were  not  all  taken  from  a  standardized  manage¬ 
ment  report,  different  levels  of  management  review  (adjustments  made  outside  and 
separate  from  direct  ALERT  calculations)  may  be  reflected  in  these  figures.  In  each 
case,  however,  the  figures  are  described  as  "ALERT”  estimates.  The  fact  that  man¬ 
agement  adjustments  are  present  in  these  different  estimates  does  not  change  the 
central  point  that  estimates  are  subject  to  massive  fluctuations.  It  does  show  that 
management  decisions  at  various  levels,  at  both  AFLC  and  the  Air  Staff,  contribute 
to  the  problem  of  fluctuating  estimates,  in  addition  to  whatever  the  ALERT  model 
does.  Successive  ALERT  estimates  of  the  FY87  last  look  were  $4.5  billion  (spring 
1984)  and  $2.98  billion  (spring  1985). 

This  level  of  instability  is  as  bad  as  that  exhibited  by  earlier  POM  esti¬ 
mating  methods  (see  Figure  2-1  in  Chapter  2),  which  means  that  ALERT  fails  to 
address  the  main  cause  of  the  credibility  problem  -  fluctuating  estimates  of  the 
same  year’s  requirement. 
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Why  does  ALERT  exhibit  this  instability?  Again,  the  volatile  behavior  of 
the  budgeting  system  and  its  central  role  in  the  ALERT  methodology  are  the  main 
reasons.  We  have  already  seen  how  the  use  of  early  D041/CSIS/transition  estimates 
as  history  undercuts  ALERT’s  chances  for  accuracy.  Use  of  these  estimates  also 
contributes  to  the  instability  problem.  When  ALERT  was  used  to  make  projections 
for  POM  87,  for  example,  second-look  estimates  (totaling  $2.7  billion)  served  as  the 
historical  FY86  last  looks  for  the  17  systems  in  ALERT.  A  year  later,  for  the 
POM  88  projections,  ALERT  was  using  fourth  looks  at  FY86  (totaling  $2.0  billion)  as 
the  FY86  last  looks.  This  $700  million  drop  in  the  ’historical”  data  is  one  reason  for 
the  change  in  the  ALERT  estimate  for  FY88,  from  $3.44  billion  in  POM  87  to 
$2.15  billion  in  POM  88. 

Another  reason  the  FY88  estimate  changed  is  that  the  FY88  forecasting 
equations  changed.  The  most  fundamental  change  was  the  introduction  of  the 
June  1985  D041/CSIS  deficit  as  a  predictor  variable  for  FY88.  In  POM  87,  FY88  was 
one  of  the  four  POM  outyears,  and  no  D041/CSIS  variable  applied.  In  POM  88,  FY88 
was  the  first  POM  year,  and  the  June  1985  D041/CSIS  variable  became  the  key 
predictor  variable.  In  general,  when  a  year  becomes  the  first  POM  year  after  having 
been  a  POM  outyear,  there  is  likely  to  be  a  significant  change  in  the  ALERT  esti¬ 
mate. 

If  the  June  D041/CSIS  estimate  is  not  used  as  the  key  predictor  variable 
for  the  first  POM  year,  ALERT  becomes  similar  to  the  POSSEM  model  developed  by 
the  Air  Staff.  POSSEM  basically  uses  only  age  and  value  of  the  fleet  as  predictor 
variables.  In  this  case,  as  demonstrated  by  the  reception  the  POSSEM  model 
received,  questions  about  the  conceptual  validity  and  reliability  of  the  model  become 
major  obstacles  to  its  acceptance. 
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CAN  OTHER  FORECASTING  METHODS  DO  BETTER? 

Lack  of  accuracy  and  stability  in  its  estimates  will  prevent  ALERT  from  solv¬ 
ing  the  credibility  problem  for  spares.  Can  other  forecasting  methods  do  better?  The 
test  of  whether  other  methods  can  do  better  is  not  whether  their  concept  and 
approach  sound  better  than  ALERT’s.  The  test  is  whether  or  not  they  will  provide 
reasonably  stable  and  accurate  estimates  of  future  spares  requirements  for  POMs. 

In  this  section,  we  take  a  brief  look  at  alternatives  to  ALERT.  Our  conclusion 
is  that  until  the  D041/CSIS/transition  budgeting  system  is  improved,  no  POM  fore¬ 
casting  method  can  do  any  better  than  ALERT. 

The  Peacetime  Operating  Stocks  Spares  Estimating  Model  (POSSEM) 

POSSEM  is  a  multilinear  regression  model  that  was  developed  by  the 
Comptroller  of  the  Air  Force  (HQ  USAF/AC).  POSSEM  preceded  ALERT  and  was 
used  by  the  Air  Force  for  POM  85  and  POM  86  projections.  The  main  difference 
between  POSSEM  and  ALERT  is  that  POSSEM  does  not  include  an  early  D041/CSIS 
estimate  in  its  set  of  predictor  variables,  but  relies  solely  upon  value  and  age  of  the 
fleet  and  (for  some  systems)  projected  utilization  rates. 

Another  important  difference  between  POSSEM  and  ALERT  is  that 
POSSEM  does  not  employ  earlier  D041/CSIS/transition  estimates  as  history  for  the 
2  years  preceding  the  first  POM  year.  Although  both  POSSEM  and  ALERT  attempt 
to  project  what  last-look  values  will  be,  POSSEM  simply  accepts  the  2-year  gap 
between  historical  last  looks  and  the  first  POM  year,  rather  than  use  estimates  of 
the  final  valuer  for  those  2  years. 

The  fact  that  D041/CSIS/transition  products  are  not  central  to  the 
POSSEM  approach  -  either  as  predictor  variables  or  as  history  substitutes  - 
means  that  POSSEM  estimates  are  likely  to  exhibit  greater  stability  over  time  than 
ALERT  estimates.  This,  in  fact,  has  been  true  of  POSSEM  estimates.  But  not  using 
D041/CSIS  inputs  hinders  POSSEM’s  acceptance  on  conceptual  grounds.  Failure  to 
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tie  in  D041/CSIS  projections  opens  POSSE M  to  the  complaint  that  it  will  not  be 
responsive  enough  to  changes  in  the  force  structure  and  flying-hour  program  - 
traditionally  viewed  as  drivers  of  future  spares  requirements.  (This  complaint  about 
POSSEM  was  raised  by  AFLC  and  was  the  primary  motivation  for  ALERT’s 
development.)  Adjusting  POSSEM  outputs  after  the  fact  to  reflect  such  changes  is 
equivalent  to  the  old  cost-per-flying-hour  method  and  will  not  solve  the  problem. 

Even  more  important  than  the  conceptual  questions  about  POSSEM  is 
the  question  of  whether  age  and  value  of  the  fleet  by  themselves  can  provide  suffi¬ 
ciently  accurate  estimates.  As  a  first  test,  we  obtained  the  POSSEM  estimate  for 
FY85,  as  it  appeared  in  POSSEM  documentation  prepared  in  January  1983  by  the 
Air  Staff.  (POM  85,  which  was  submitted  in  May  1983,  was  prepared  using 
POSSEM. )8  The  POSSEM  estimate  for  FY85  was  $3.4  billion,  plus  an  estimate  of 
$705  million  in  unfunded  carry-over,  for  a  total  of  $4.1  billion.  The  true  last  look  at 
FY85  was  $2.3  billion.  POSSEM  missed  by  $1.8  billion  -  a  78-percent  error. 

As  a  second  test  (motivated  by  the  fact  that  a  programming  error  was  dis¬ 
covered  in  POSSEM  in  April  1985,  which  cast  doubt  on  earlier  projections),  we  did 
our  own  projection  of  FY85,  as  we  did  for  ALERT,  but  this  time  following  the 
POSSEM  approach.  Using  historical  last  looks  by  system  for  the  years  FY78 
through  FY82  as  the  observed  values  of  the  dependent  variable,  regressions  were 
done  using  value  and  the  reciprocal  of  age  as  the  two  independent  variables  for  each 
of  the  17  systems  of  interest,  From  the  derived  relations,  we  projected  the  FY85  last 
look,  using  the  value  and  age  amounts  that  applied  in  FY85.  When  the  results  were 
added,  the  aggregate  projection  for  FY85  was  $5.5  billion.  The  true  FY85  last  look 
was  $2.3  billion.  Thus,  in  our  test  of  the  POSSEM  approach,  the  method  missed  the 


^Colonel  Donald  G.  Kane,  Director  of  Cost  and  Manugoment  Anulysls  (HQ  L'SAF/AC.Vt), 
Study  of  Air  Force  Aircraft  Replenishment  Spare  Requirements,  (Washington,  D  C,:  HQ  L’SAF/ 
ACM,  January  1983), 
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desired  answer  by  $3.2  billion  -  a  140-percent  error.  The  Appendix  gives  additional 
details  about  the  test. 

On  the  basis  of  these  results,  combined  with  the  POSSE  M’s  problems  in 
gaining  conceptual  acceptance,  we  do  not  think  POSSEM  or  POSSEM  variants 
would  be  any  more  successful  than  ALERT  in  solving  the  credibility  problem  for 
spares  requirements. 

MACROSTRAT 

MACROSTRAT  represents  a  totally  different  approach  to  the  spares  fore¬ 
casting  problem  for  POMs.  The  basic  idea  is  to  define  an  average  component  for  each 
weapon  system  in  terms  of  average  pipeline  factors:  failure  rates,  repair  times,  and 
condemnation  rates.  Then,  for  this  average  item,  a  deterministic  calculation  is  per¬ 
formed  to  compute  unit  requirements  for  the  budget  year  and  the  POM  years.  The 
unit  price  for  the  item  is  not  the  average  unit  price  across  all  items  used  on  the 
system,  but  rather  is  set  so  that  the  budget-year  requirement  for  the  item  times  this 
price  is  equal  to  the  computed  D041/CSIS/transition  budget  requirement  across  all 
items  for  the  system.  MACROSTRAT  computes  the  dollar  requirement  for  the  POM 
by  multiplying  the  unit  requirement  for  each  POM  year  by  this  "calibrated”  unit 
price. 

The  MACROSTRAT  approach  does  distinguish  between  net  and  gross 
requirements  and,  for  that  reason,  is  an  improvement  over  cost-per-flying-hour 
methods.  Obviously,  however,  changes  in  item  factors  from  year  to  year  in  the  D041 
data  base,  including  changes  in  the  value  of  the  calibrated  unit  price,  will  influence 
MACROSTRAT  forecasts.  Thus,  in  its  own  way,  MACROSTRAT  is  as  dependent  on 
the  underlying  budgeting  system  as  ALERT. 

Quantitative  tests  of  the  MACROSTRAT  approach  were  not  done  for  this 
study.  MACROSTRAT-like  approaches,  therefore,  cannot  be  categorically  dismissed 
as  having  no  promise.  It  is  possible  that  such  approaches,  perhaps  tailored  to 
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different  categories  of  spare  parts  (e.g.,  avionics,  hydraulics,  and  mechanical)  are 
worth  exploring.  But  the  Air  Force  must  keep  in  mind  that  the  budgeting  system 
itself  is  the  root  of  the  spares  forecasting  problem,  and  efforts  that  focus  on  POM 
forecasting  run  the  risk  of  diverting  needed  attention  and  resources  from  the  under¬ 
lying  budgeting  problem. 

How  the  Army  and  Navy  Do  It 

Because  of  the  attention  Air  Force  problems  have  received,  the  perception 
may  be  that  the  Air  Force  is  the  only  Service  having  difficulties  forecasting  aircraft 
spares  requirements.  In  fact,  both  the  Army  and  the  Navy  have  had  similar  pro¬ 
blems. 

The  Army,  using  the  Requirements  Determination  and  Execution  System 
(RD/ES)  within  the  Army  Materiel  Command's  Commodity  Command  Standard 
System  (CCSS),  computes  spares  requirements  with  item-specific,  bottom-up,  0041- 
Uke  methods.  A  key  feature  distinguishing  the  Army  from  the  Air  Force  is  that  the 
Army  pushes  item  calculations  through  the  entire  POM  period.  Like  the  Air  Force, 
the  Army  makes  transition  adjustments  to  computed  requirements  to  obtain  the 
final,  official  requirement. 

The  item-specific,  bottom-up  approach  has  not  made  the  Army  more  suc¬ 
cessful  than  the  Air  Force  in  forecasting  spares  requirements.  Between  FY80  and 
FY84,  the  Army's  total  requirement  for  aircraft  spares  (initial  provisioning,  replen¬ 
ishment,  and  war  reserve)  increased  substantially  -  from  approximately  $70  mil¬ 
lion  to  more  than  $600  million.  Like  the  Air  Force,  the  Army  discovered  that  its 
spares  requirements  system  was  failing  to  give  reliable  and  accurate  forecasts  of 
future  requirements.  The  similarity  of  the  Army’s  and  Air  Force’s  problems  is 
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indicated  by  the  fact  that  in  the  fall  of  1983,  the  Army  commissioned  a  major  study9 
of  aircraft  spare  parts  requirements  forecasting,  a  study  very  similar  in  spirit  to  the 
CORONA  REQUIRE  study  by  the  Air  Force  a  year  earlier. 

A  finding  of  the  Army  study  was  that  POM  and  budget  forecasts  of  a 
given  year’s  requirement  tended  to  be  very  different  from  actual  "end-of-year” 
requirements.  As  an  example,  the  study  cites  POM  and  budget  forecasts  for  FY83 
that  "amounted  to  only  44  percent  and  48  percent  respectively  of  the  eventual  FY83 
end-of-year  actual  requirement."  The  study  goes  on  to  state  that  "forecast  errors  of 
this  magnitude  dearly  exceed  that  which  should  be  considered  acceptable  for  budget 
forecasting  purposes." 

The  principal  recommendation,  of  the  Army  study  was: 

The  Army,  specifically  HQDA,  should  make  a  conscious  decision  that 
the  status  quo  in  the  aircraft  spare*  program  and  budget  requirements 
determination  process  is  no  longer  acceptable  and  commit  resources  to  ini¬ 
tiate  and  sustain  both  shor  .-term  and  long-term  corrective  actions 

In  its  continuing  efforts  to  solve  these  problems,  the  Army  has  not 
attempted  to  develop  alternative  POM  forecasting  methods  to  replace  its  item- 
specific  approach.  Instead,  mainly  as  a  result  of  influence  and  control  exerted  by  the 
Vice  Chief  of  Staff’  (HQDA|,  the  Army  has  recognized  the  need  for  stability  in 
requirements  estimates  and  has  instituted  management  and  review  procedures  at 
Headquarters  to  enforce  greater  discipline  in  the  POM-to-budget-to-executior 
process  for  spares. 

Like  the  Air  Force  and  the  Army,  the  Navy  has  seen  its  requirements  for 
aircraft  spares  grow  in  recent  years.  Also  like  the  Air  Force  and  Army,  the  Navy’s 
has  a  bottom-up,  Item-specific  computational  system  for  computing  budget  and 

9Fred  J.  Lokay,  et  al . ,  Aircraft  Spare  Parts  Requirements  Forecasting  Study,  Final  Report, 
Report  1404-04-84-CR,  prepared  for  the  Office  of  the  Deputy  Chief  of  Staff  (Research,  Development 
and  Acquisition),  [HeadquarVjis,  Department  of  the  Army  (hv>DA):  General  Research  Corpora¬ 
tion!,  October  1984. 


execution  requirements  for  spares  -  the  C  yclic  Forecasting  and  Levels  Program 
(D01),  which  is  embedded  in  the  Navy’s  mechanized  Uniform  Inventory  Control 
Program  (UICP)  system.  What  distinguishes  the  Navy’s  approach  is  the  greater  role 
that  transition  adjustments  play  in  determinirg  final  requirements  -  after  item- 
specific  computations  have  been  completed. 

The  major  role  played  in  the  final  requirement  by  program- related  transi¬ 
tion  adjustments  makes  the  Navy's  POM  forecasting  problem  somewhat  different  in 
character  than  the  Air  Force’s.  Given  the  greater  emphasis  on  program-related 
requirements,  POM  "forecasts”  in  the  Navy  tend  to  be  more  statements  of  planned 
programs  than  predictions  of  what  the  item-specific  requirements  system  will 
produce.  These  "forecasts”  are  "successful”  to  the  extent  that  the  Navy  sticks  to  its 
plan  -  either  by  proceeding  with  planned  programs  or  by  reallocating  resources 
(without  changing  the  total)  as  plans  change. 

The  structure  of  a  typical  Navy  budget  in  comparison  with  an  Air  Force 
budget  illustrates  the  greater  role  played  by  program-type  requirements  in  the 
Navy.  In  the  Navy’s  aircraft  spares  replenishment  budget  request  for  FY84,  line  18 
of  the  P-13  transition  document  shows  the  net  FY84  requirement  calculated  by  the 
Navy’s  item-specific  computational  and  stratification  system  to  be  $1.48  billion. 
There  are  then  32  different  columns  for  additional  positive  and  negative  adjust¬ 
ments  to  stratification  results,  with  a  final  net  requirement  statement  of  $4.07  bil¬ 
lion. 

These  adjustments  include:  $622.5  million  in  initial  spares  requirements 
not  included  in  stratification  (the  Navy  includes  certain  initial  spares  requirements 
in  the  replenishment  request),  $901.5  million  in  follow-on  requirements  "not  in 
stratification”  (NIS),  $161.4  million  in  Aviation  Coordinated  Allowance  List 
(AVCAL)  maintenance,  and  $82.3  million  in  F-18  rep’enishment  NIS.  Those 
program-type  adjustments  are  in  addition  to  standard  adjustments  for  such  items  as 


carry-over  from  prior  years  ($709.5  million)  and  price  redetermination  and  escala¬ 
tion  ($87.1  million). 

In  comparison,  the  Air  Force  budget  request  for  FY84  shows  a  computed 
D041/CSIS  deficit  of  $1.37  billion,  which,  after  transition  adjustments,  becomes 
$3.34  billion.  Although  the  magnitude  of  the  change  is  comparable  with  that  in  the 
Navy  budget,  the  reasons  are  different.  The  difference  in  the  Air  Force  budget  is 
driven  by:  $281.3  million  in  scrub  adjustments;  $260.1  million  in  new,  additive 
programs;  $619.3  million  due  to  price  redetermination;  and  $904.5  million  in 
unfunded  carry-over  requirements.  Although  program-type  adjustments  and  addi¬ 
tives  contribute,  they  are  not  as  influential  in  the  Air  Force  budget  as  in  the  Navy 
budget. 

Of  course,  the  Navy’s  greater  reliance  on  program-type  requirements  does 
not  in  itself  guarantee  that  the  Navy’s  total  requirements  will  remain  stable  from 
POM  to  budget  to  execution.  For  that  to  happen,  planning  and  funding  discipline 
must  be  enforced.  Although  there  are  no  formal  mechanisms  in  the  Navy  require¬ 
ments  system  that  force  program  managers  to  make  tradeoffs,  the  Navy,  like  the 
Army,  has  developed  operating  procedures  that  promote  this  discipline.  Conversa¬ 
tions  with  former  Naval  Supply  System  Command  (NAVSUP)  personnel  indicate 
that,  in  the  early  1980’s,  senior  NAVSUP  management  (like  senior  Army  manage¬ 
ment)  began  insisting  upon  consistency  over  time  in  the  programming,  budgeting, 
and  execution  process  for  spares. 

The  Navy  does  have  a  macro-level  forecasting  procedure  for  predicting 
the  dollar  value  of  the  recurring-demand-based  portion  of  spares  requirements  in  the 
POM  years.  Called  the  Value  of  Operating  Aircraft  (VOAC)  method,  the  Navy  mul¬ 
tiplies  the  dollar  vdue  of  the  recurring-demand-based  portion  of  the  current  (e.g., 
budget)  requirement  by  the  ratio  of  the  dollar  value  of  operating  aircraft  in  the  POM 
years  to  the  current  dollar  value  of  operating  aircraft.  Although  not  a  regression, 
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the  VOAC  method  is  similar  in  concept  to  ALERT  and  POSSEM  in  relating  spares 
requirements  to  the  value  of  aircraft.  Because  recurring-demand-based  require¬ 
ments  represent  only  a  small  portion  of  the  final  requirement,  however,  the  stability 
and  predictive  accuracy  of  the  Navy's  method  are  not  as  visible  or  as  crucial  to  the 
Navy’s  spares  program  as  to  the  Air  Force’s.  With  ALERT,  the  Air  Force  attempts  to 
project  the  entire  requirement,  including  program-reiated  requirements.  The  Navy 
separates  program- related  requirements,  defends  them  as  such,  and  follows  up  with 
deliberate  management  review  that  is  aimed  specifically  at  preserving  reasonable 
consistency  in  the  spares  requirement  over  time. 

In  effect,  the  Navy’s  approach  recognizes  that  attempts  to  forecast 
program-type  requirements,  which  represent  a  substantial  portion  of  the  total 
requirement,  are  futile.  Program  requirements  are  requirements  that  will  be  deter¬ 
mined  largely  by  future  management  decisions  -  decisions  that,  over  POM  Fore¬ 
casting  hurizons,  cannot  be  foreseen  by  technical  forecasting  models. 

Neither  the  Army  nor  the  Navy,  therefore,  has  developed  a  technical 
forecasting  tool  that  solves  the  spares  forecasting  problem  for  POMs.  Instead,  both 
Services  have  reacted  to  the  problem  with  management  initiatives  designed  to  yield 
greater  consistency  in  the  programming,  budgeting,  and  execution  process. 

Other  Methods  ? 

Is  it  possible  that  some  other  approach,  different  from  the  ones  we  have 
discussed,  might  work?  Regardless  of  what  technique  is  used  to  make  POM  fore¬ 
casts,  the  same  basic  question  must  be  asked:  Will  it  solve  the  credibility  problem 
for  spares? 

We  have  argued  that  bad  budget  fortcasts,  not  POM  forecasts,  are  the. 
main  cause  of  the  credibility  problem.  No  POM  forecasting  model,  therefore,  can 
solve  the  credibility  problem.  Regardless  of  how  POM  forecasts  are  developed,  DoD 
policy  and  established  procedures  [such  as  Central  Secondary  Item  Stratification 


(CSIS/3  are  such  that  budgets  will  continue  to  be  put  together  with  item-specific 
computational  systems.  As  long  as  projections  made  with  those  systems  are  flawed, 
the  credibility  problem  will  persist. 

The  positive  side  is  that  if  the  budgeting  system  is  improved,  POM  fore¬ 
casts  will  improve.  The  results  presented  earlier,  showing  that  the  ALERT  approach 
works  reasonably  well  when  real  history  is  used  instead  of  estimates,  suggest  this  is 
true. 
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5.  MANAGEMENT,  NOT  FORECASTING,  IS  THE  ANSWER 


We  have  seen  that,  since  the  early  1980's,  neither  the  D041/CSIS/transition 
system  nor  POM  forecasting  models  built  around  that  system  have  been  able  to 
provide  the  Air  Force  with  reliable  estimates  of  future  spares  requirements.  With 
continuing  change  in  the  overall  Air  Force  program,  this  problem  is  likely  to  persist. 
No  matter  what  their  form,  forecasting  techniques  by  themselves  can  never  guar¬ 
antee  the  correctness  of  statements  of  future  spares  requirements. 

To  solve  this  problem,  the  Air  Force  needs  to  move  beyond  reliance  on  fore¬ 
casting  alone  and  recognize  that  control  of  the  aggregate  requirement,  through 
management,  is  also  necessary  to  ensure  consistency  from  POM  to  budget  to 
execution.  By  instituting  procedures  deliberately  aimed  at  stabilizing  the  total 
requirement,  the  Air  Force  can  achieve  through  management  what  it  has  been 
unable  to  accomplish  through  forecasting:  credible,  defensible  statements  of  ftiture 
spares  requirements. 

For  both  AFLC  and  the  Air  Staff,  the  idea  that  the  requirements  system  should 
be  used  to  manage  and  control  requirements,  in  addition  to  computing  them,  repre¬ 
sents  a  new  view  of  the  system.  In  the  traditional  view,  the  requirement  -  whether 
it  is  today’s  execution  requirement  or  tomorrow’s  budget  and  POM  -  is  whatever 
the  requirements  system  says  it  is.  The  new  view  recognizes  that  budgets  and  POMs 
reflect  human  judgments  and  decisions,  which  may  be  informed  by  computer-based 
estimates  but  cannot  and  should  not  be  determined  by  them.  Budgets  are,  after  all, 
just  that  -  budgets',  changes  in  computed  material  requirements  are  not  always 
automatically  sufficient  to  justify  departing  from  the  funding  constraints  that 
budgets  impose.  Until  the  Air  Force  recognizes  this  and  begins  to  exercise  better 


management  control  of  the  many  decisions  that  go  into  the  requirement,  spares 
requirements  will  continue  to  be  troublesome. 

The  challenge,  of  course,  is  to  exercise  control  of  the  aggregate  funding  require¬ 
ment  without  sacrificing  the  flexibility  needed  to  deal  with  changing  material 
requirements.  To  meet  this  challenge,  the  Air  Staff  should  fully  acknowledge  that 
material  requirements  will  always  change  from  POM  to  budget  to  execution  and 
should  be  prepared  to  explain  these  changes  to  decision-makers  in  the  programming 
and  budgeting  process.  AFLC,  for  its  part,  should  be  prepared  to  use  the  require¬ 
ments  system  as  an  allocation  tool,  to  reallocate  resources  as  necessary  when 
changes  occur,  so  that  the  total  funding  requirement  can  be  held  reasonably  stable. 

Two  prerequisites  apply  if  AFLC  and  the  Air  Staff  are  to  do  this:  Managers  at 
all  levels  must  be  able  to  obtain  accurate,  up-to-date  information  on  spares  require¬ 
ments  by  weapon  system,  and  AFLC  and  the  Air  Staff  need  to  improve  their  under¬ 
standing  of  each  other’s  jobs. 

Information  by  weapon  system  is  important  because  managers  need  to  know 
when  system  requirements  have  changed  and  how  to  reallocate  resources  to  deal 
with  that  change.  Better  understanding  between  AFLC  and  the  Air  Staff  is  impor¬ 
tant  because  of  the  different  demands  the  two  organizations  place  on  the  require¬ 
ments  system.  AFLC  needs  a  system  that  tracks  changing  requirements;  the  Air 
Staff  needs  one  that  produces  stable  POM  and  budget  estimates.  Unless  the  two 
organizations  recognize  this  fact  and  work  together,  it  will  be  very  difficult  for  any 
requirements  system  to  meet  their  conflicting  demands. 


Providing  information  on  spares  requirements  by  weapon  system  is  one  of  the 
central  goals  of  the  Requirements  Data  Bank  (RDB)  project  now  underway  at  AFLC. 
The  RDB  project  represents  a  significant  AFLC  effort  to  modernize,  integrate,  and 
improve  peacetime  and  wartime  requirements  and  execution  systems  in  the  Air 


Force  for  reparable  and  consumable  secondary  items,  support  equipment,  and 
engines.  The  RDB  project  has  two  primary  objectives: 

1.  To  develop  forecasts  of  logistics  resources  required  to 
achieve  specific  weapon-system  readiness  and  sustain¬ 
ability  goals  throughout  the  POM  period 

2.  To  develop  a  logistics  operations  and  budget  execution 
system  to  achieve  the  weapon-system  support  goals  that 
have  been  resourced  in  the  POM  process  J 

Oiven  the  importance  of  managing  the  aggregate  spares  requirement,  we 
recommend  the  Air  Force  adopt  the  following  as  an  integral  part  of  the  second 
objective: 


The  RDB  will  serve  as  a  tool  enabling  logistics  managers  to 
review,  control,  and  prioritize  spares  requirements  as  they 
develop,  and  reallocate  resources  as  necessary.  This  will  ensure 
system-level  flexibility  and  responsiveness  for  purposes  of 
execution,  while  at  the  same  time  making  stability  at  the 
aggregate  level  possible  for  purposes  of  programming  and  bud¬ 
geting. 

The  RDB  project  is  aimed  at  improving  the  content  and  accessibility  of  the 
spares  data  base  through  development  of  improved  computational  and  data  process¬ 
ing  capabilities.  Improved  capabilities  are  only  half  the  story,  however.  Unless  the 
Air  Force  takes  advantage  of  the  RDB’s  new  capabilities  and  useB  them  to  manage 
spare s  requirements  —  not  just  compute  and  report  them  —  the  RDB  will  be  no  more 
successful  than  existing  systems  have  been  in  solving  the  credibility  problem  for 
spares. 

AFLC  AND  THE  AIR  STAFF 

AFLC  and  the  Air  Staff  have  different  jobs.  AFLC’s  prime  job  is  execution. 
Most  of  AFLC’s  attention  and  effort  is  spent  preparing  the  short-term  plans  and 


IThe  Air  Force  Requirement «  Data  Bank  (RDB):  Motivation,  Objectives  and  Development 
Strategy,  Colonel  Robert  S.  Tripp,  RDB  Program  Manager,  Logistics  Management,  SystemB  Center, 
Air  Force  Logistics  Command;  and  Mb,  Diann  LuwBon,  Assistant  for  Requirements  Development, 
Directorate  of  Materiel  Requirements  (Wrlght-Putterson  AFB,  Ohio:  Headquarters,  Air  Force 
LogistlcB  Command,  March  1988), 
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making  the  day-to-day  procurement,  repair,  and  transportation  decisions  necessary 
to  provide  the  Air  Force  with  continuing  logistics  support.  To  do  that  job  well,  AFLC 
needs  a  dynamic  requirements  system  that  tracks  and  responds  to  change. 

Air  Staff  offices  concerned  with  spares  requirements,  on  the  other  hand,  have 
the  task  of  presenting  and  defending  programs  and  budgets  in  the  annual  Planning, 
Programming,  and  Budgeting  System  (PPBS)  process.  To  do  this,  the  Air  Staff  relies 
on  the  requirements  system  to  provide  stable  and  accurate  estimates  of  future 
requirements.  If  responsiveness  in  the  AFLC  system  causes  budget  estimates  to 
change,  the  Air  Staffs  job  becomes  very  difficult.  Rather  than  changing  the  budget, 
it  is  better  to  use  system  responsiveness  to  reallocate.  Application  of  the  require¬ 
ments  system  to  the  task  of  monitoring  and  controlling  requirements,  in  addition  to 
computing  them,  is  necessary  for  the  Air  Force  to  live  within  its  budget. 

For  the  Air  Staff  and  AFLC  to  work  together,  therefore,  two  points  are 
important.  The  Air  Staff  must  fully  acknowledge  that  requirements  for  recoverables 
are  inherently  subject  to  change  and  that  AFLC  heeds  a  system  that  tracks  this 
change  to  do  a  good  job  in  execution.  AFLC,  for  its  part,  should  recognize  that  the 
Air  Staff  needs  reasonably  stable  estimates  of  future  requirements  if  spares  funding 
requests  are  to  be  defended  successfully  in  the  programming  and  budgeting  process. 
Otherwise,  it  will  be  very  difficult  for  AFLC  and  the  Air  Staff  to  set  up  the  joint 
management  mechanisms  required  for  effective  use  of  the  RDB. 

THE  STOCK-FUNDING  OPTION 

For  other  reasons,  apart  fVom  the  POM  forecasting  problem,  the  Air  Force  is 
considering  the  possibility  of  stock  funding  as  an  alternative  way  to  finance  Inven¬ 
tories  of  reparable  spares.  Aviation  reparables  became  stock-funded  in  the  Navy  in 
FY85,  and  both  the  Air  Force  and  Army  are  studying  the  possibility  of  following 
suit  -  partly  in  response  to  Congressional  suggestions.  If  reparables  are 


stock-funded  in  the  Air  Force,  there  are  several  implications  for  the  POM  forecasting 
problem. 

Under  the  present  system,  the  Air  Force’s  supply  system  replenishes,  renews, 
and  builds  its  stocks  of  reparable  spares  with  procurement  funds  appropriated  each 
year  by  Congress.  These  spares  are  issued,  at  no  charge,  to  users  at  Air  Force  bases 
and  depots  as  needed.  Under  stock  funding,  the  supply  system  would  continue  to 
draw  on  appropriated  funds  to  build  new  inventories  to  support  force  growth,  mod¬ 
ernization,  and  readiness  improvements  (in  the  stock  fund  this  is  called  "inventory 
augmentation”),  but  the  system  would  begin  using  nonappropriated  funds  from  a 
revolving  cash  account,  the  Air  Force  stock  fund,  to  meet  replenishment  require¬ 
ments  (e.g.,  replacement  of  condemnations  and  requirements  caused  by  churn). 

POMs  are  not  required  or  submitted  for  stock-fund  peacetime  replenishment 
requirements  -  only  budgets  are  submitted.  Thus,  if  reparables  are  stock-funded,  it 
will  no  longer  be  necessary  to  make  POM  forecasts  of  peacetime  replenishment 
requirements  for  spares.  This  is  one  of  the  reasons  the  Navy  adopted  stock  funding 
for  reparables. 

As  noted  above,  though  POMs  are  not  required  under  stock  funding,  budgets 
still  must  be  prepared.  Even  here,  however,  there  is  an  important  difference.  Stock- 
fund  replenishment  budgets  are  not  subject  to  the  Congressional  review  that  is 
applied  to  appropriated  budgets  (such  as  Budget  Program  1500  for  spares  in  the  Air 
Force).  Final  review  and  approval  authority  for  stock-fund  replenishment  budgets 
lies  with  the  OSD  Comptroller  and  the  Office  of  Management  and  Budget.  This 
makes  the  approval  and  adjustment  process  easier  for  stock-fund  budgets  than  for 
appropriated  budgets. 

Under  stock  funding,  POM  forecasts  would  still  be  required  for  inventory  aug¬ 
mentation  requirements.  Most  of  these  requirements,  however,  are  program-based 
rather  than  recurring-demand-based  requirements.  POM  "forecasts"  of  inventory 
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augmentation  requirements,  therefore,  could  be  presented  as  statements  of  specific 
program  requirements  (the  way  the  Navy  does  it),  rather  than  as  forecasts  (as 
ALERT  presents  them). 

Oiven  the  difficulty  of  projecting  program-type  requirements  with  historically 
based  forecasting  models,  the  fact  that  stock  funding  would  force  the  Air  Force  to 
distinguish  between  recurring-demand-based  requirements  and  program-based  aug¬ 
mentation  requirements  is  a  good  thing.  With  improved  visibility  of  what  is 
program-based  and  what  is  not  (a  place  where  the  RDB  should  help),  it  would  be 
easier  to  exercise  management  control  of  requirements.  The  distinction  would  also 
make  it  easier  to  defend  requirements  in  the  programming  and  budgeting  process  — 
both  within  the  Air  Force  and  before  OSD  and  Congress. 

Although  stock  funding  eliminates  one  POM  forecasting  problem,  it  creates 
another.  The  stock  fund  replenishes  itself  through  sales  of  stock-funded  material  to 
Air  Force  and  other  DoD  and  government  customers,  who  generally  pay  for  their 
purchases  with  funds  from  either  appropriated  Operations  and  Maintenance  (O&M) 
accounts,  or  -  in  the  case  of  depot  customers  -  industrial  fund  accounts.  POMs 
must  be  prepared  for  many  of  these  accounts.  Thus,  under  stock  funding  of 
reparables,  it  will  be  necessary  to  make  POM  forecasts  of  the  funding  required  in 
customer  accounts  (e.g.,  the  O&M  appropriation)  to  enable  customers  to  buy  the 
spares  they  need  from  the  stock  fund. 

However,  because  customers  need  funding  when  their  material  requirements 
exist,  POM  estimates  for  customer  accounts  could  be  made  later  than  present  POM 
estimates  for  appropriated  procurement  accounts.  This  means  that  POM  estimates 
can  be  made  closer  in  time  to  the  point  when  material  requirements  exist.  Short¬ 
ening  the  POM  forecasting  horizon  in  this  way  improves  the  chances  of  correctly 
predicting  requirements  and  correctly  allocating  resources  to  meet  them.  Figure  6-1 
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illustrates  the  shortening  effect  in  relation  to  FY88,  if  reparables  were  already 
stock-funded. 


FIGURE  5-1.  POM  FORECASTING  IP  REPARABISS  ARE  STOCK-FUNDED 


The  time  line  in  Figure  5-1  is  the  same  as  that  in  Figure  2-2  in  Chapter  2.  As 
noted  in  Chapter  2,  under  the  present  system,  the  POM  forecast  for  FY88  must  be 
made  in  the  fall  of  1985.  Funding  requirements  in  FY88  correspond  to  materiel 
requirements  a  procurement  leadtime  later.  Since  the  average  procurement  lead- 
time  for  reparables  is  2  years,  FY88  funding  requirements  correspond  to  material 
requirements  in  FY90. 

Figure  5-1  shows  that  under  stock  funding,  there  is  no  peacetime  replenish¬ 
ment  POM  required  for  FY88;  the  first  POM  that  must  be  prepared  in  connection 
with  FY90  material  requirements  is  POM  90,  which  is  a  full  procurement  leadtime 
closer  to  FY90  than  POM  88.  The  first  forecast  of  FY88  funding  requirements  that 


must  be  prepared  is  the  FY88  budget.  Furthermore,  the  PPBS  calendar  is  such  that 
the  POM  90  forecast  is  submitted  after  the  FY88  stock-fund  budget  has  been 
assembled.  This  means  that  if  the  FY88  budget  is  a  good  forecast  of  what  the  stock 
hind  needs  to  place  on  order  in  FY88  to  meet  material  requirements  in  FY90,  POM 
60  can  use  the  FY88  budget  as  the  basis  for  its  estimate  of  customer  funding 
requirements  in  FY90.  ^  ^ 

For  the  outyears  of  POM  90  (FY91  through  FY94),  the  POM  forecasting 
problem  for  customer  accounts  under  stock  funding  is  very  similar  to  the  one  under 
the  present  system.  Thus,  stock  funding  does  not  eliminate  the  need  for  continued 
work  on  long-range  forecasting  techniques  for  spares.  In  terms  of  numbers  that  have 
the  most  effect  on  resource  allocation  in  the  Pentagon,  however,  estimates  for  the 
POM  outyears  are  not  as  important  as  those  for  the  first  POM  year.  If  forecasts  for 
the  first  POM  year  can  be  improved  under  stock  funding,  it  should  be  possible  to 
make  better  outyear  forecasts,  perhaps  with  ALERT  or  other,  simpler  methods. 

The  final  and  most  important  point  about  stock  funding  iB  this:  Stock  funding 
affects  only  the  POM  forecasting  problem.  Stock  funding  does  not  alter  the  fact  that 
there  is  a  budget  forecasting  problem,  or  reduce  its  severity.  For  example,  if 
reparables  were  already  stock-funded  in  1986,  no  replenishment  POM  would  have 
been  necessary,  but  the  Air  Force  would  still  have  had  to  make  a  budget  estimate  of 
the  FY88  funding  requirement  (see  Figure  5-1).  The  only  advantage  to  be  gained 
from  stock  funding  is  that  by  reducing  the  size  and  difficulty  of  the  POM  forecasting 
problem,  the  Air  Force  would  be  able  to  apply  more  attention  and  resources  to  the 
budgeting  problem. 

SUMMARY 

With  the  advent  of  the  RDB  and  the  possibility  that  reparables  will  become 
stock-funded,  the  spares  forecasting  problem  in  the  Air  Force  will  become  easier. 
The  RDB  will  provide  information  that  will  enable  the  Air  Force  to  better 


understand  how  and  why  spares  requirements  change,  and  this  will  help  the  Air 
Force  make  better  forecasts.  Stock  funding  of  reparables,  if  it  happens,  will  shorten 
forecasting  leadtimes  and  reduce  the  size  and  difficulty  of  the  POM  problem,  allow¬ 
ing  the  Air  Force  to  concentrate  on  the  budget  forecasting  problem  for  spares. 

In  any  case,  successful  spares  forecasting  is  more  likely  to  be  achieved  through 
better  cooperation  between  AFLC  and  the  Air  Staff  in  developing,  prioritizing,  and 
controlling  spares  requirements  than  through  development  of  increasingly  more 
complicated  forecasting  techniques. 


APPENDIX 


THE  D041  CALENDAR  AND  FURTHER  DETAILS 
ON  THE  ANALYSES  OF  ALERT  AND  POSSEM 

This  appendix  presents  some  additional  technical  details  concerning  the  D041 
calendar  and  additional  explanation  of  the  statistical  analyses  done  of  the  ALERT 
and  POSSEM  mode1*. 

The  meaning  of  the  terms  CY,  AY,  BY,  and  EY  (current  year,  apportionment 
year,  budget  year,  and  extended  year)  in  conjunction  with  the  D041  system  can  be 
understood  using  Figure  A-l.  In  the  forecasting  timeline  in  Figure  A-l,  the  last 
quarter  of  FY85  is  the  CY,  and  the  years  FY86,  FY87,  and  FY88  are  the  AY,  BY,  and 
EY,  respectively.  The  June  1985  D041/CSIS  computation  is  the  first  item-level  com¬ 
putation  that  can  see  F Y88  funding  requirements.  This  is  illustrated  by  the  vertical 
arrow  on  the  left  side  of  Figure  A-l,  which  indicates  that  the  June  1985  D041/CSIS 
computation  serves  as  input  to  the  ALERT  projection  for  POM  88.  Note  that  the 
June  computation  is  not  scrubbed  (i.e.,  subjected  to  transition).  The  September  1985 
computation  is  scrubbed  and  sees  FY88,  but  its  products  are  not  available  to  ALERT 
in  time  to  meet  AFLC’s  fall  deadline  for  delivering  POM  estimates  to  the  Air  Staff. 

Note  also  that  there  is  no  D041/CSIS  product  that  provides  information  on 
funding  requirements  in  the  4  outyears  of  the  POM.  All  of  those  years  are  beyond 
the  extended  year,  and  the  D041/CSIS  system  simply  does  not  look  that  far  ahead  for 
funding  requirements.  This  is  one  reason  why  AFLC  has  no  choice  but  to  seek  mucro 
(i.e.,  non-item-specific)  projection  methods  for  POMs.  Another,  more  important 
reason  is  that  item-specific  projections  are  not  reliable  that  far  into  the  future  -  as 
discussed  in  Chapter  3. 


ALERT  docs  not  employ  regression  at  the  aggregate  level.  Aggregate 
estimates  are  obtained  by  addition  of  the  17  estimates  obtained  from  ALERT’s 
system-level  regressions.  ALERT  developers  have  suggested  that  the  model’s 
aggregate  precision  is  better  than  its  precision  at  the  system  level,  but  they  have 
presented  no  quantitative  proof.  The  aggregate  prediction  intervals  presented  in 
Chapter  4  show  that,  indeed,  ALERT'S  aggregate  precision  is  better.  The  following 
explains  how  aggregate  prediction  intervals  were  computed. 

Each  system-level  regression  produces  an  estimated  mean  value  for  the 
system’s  requirement  and  a  coefficient  of  determination  (r2)  that  measures  the 
strength  of  the  regression  relationship.  The  estimated  mean  is  the  mean  for  a 
t-distribution,  with  a  variance  that  can  be  computed  from  the  r2  value,  and  degrees 
of  freedom  based  on  the  number  of  variables  used  in  the  regression.  Prediction 
intervals  at  the  system  level,  like  the  one  shown  for  the  C-5  in  Figure  4-3,  are 
confidence  intervals  in  these  t-distributions. 


From  the  individual  distributions,  aggregate  prediction  intervals  can  be 
constructed  through  simulation.  The  intervals  cited  in  Chapter  4  were  constructed 
by  making  1,000  draws  from  each  of  the  17  t-distributions  defined  in  ALERTs  FY88 
projections  for  POM  88.  Draws  were  made  using  the  STATGRAPHICS  (STSC 
Corporation,  Rockville,  Maryland)  statistical  package  operating  on  an  IBM  personal 
computer.  Adding  the  results  of  the  draws  across  the  17  systems  gave  1,000  values 


from  the  unknown,  composite  distribution  of  the  aggregate  requirement.  Sorted, 
these  1,000  values  define  an  empirical  histogram  of  the  aggregate  distribution. 
Aggregate  prediction  intervals  were  obtained  from  the  sorted  list  by  counting  the 
number  of  values  between  the  bounds  of  the  desired  interval  and  dividing  by  1,000  to 
obtain  the  confidence  level. 

Regarding  the  test  of  the  POSSEM  approach  described  in  Chapter  4,  there  are 
technical  problems  in  the  POSSEM  method  that  require  special  treatment.  In 
particular,  for  the  system  category  of  common  components,  the  age  and  value  of  the 
"fleet"  are  not  well-defined,  and,  for  the  system  of  "other  systems,”  age  is  not  well- 
defined.  Following  the  POSSEM  approach  in  these  two  cases,  regressions  were 
performed  only  against  value  for  "other  system.”  For  common  components,  system- 
by-system  regressions  were  performed,  using  common  component  percentages  of 
each  weapon  system’s  requirement  as  the  value  of  the  dependent  variable.  The 
common  component  breakout  percentages  were  taken  from  POSSEM  documenta¬ 
tion. 

In  the  simulated  POSSEM  projections,  the  predicted  values  for  the  C  5  and  the 
F- 15  each  exceeded  by  more  than  $1.0  billion  the  actual  "last  look”  for  those  systems. 
If  we  remove  these  two  systems  from  consideration,  the  actual  last  look  becomes 
$2.16  billion,  and  the  simulated  POSSEM  projection  is  $3.24  billion.  The  POSSEM 
approach  still  makes  a  $1.21  billion  (51  percent)  error. 
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